+ PROC_LSST_SUBMIT_PATH=/mmfs1/home/stevengs/dirac/DEEP/submit + bps submit /gscratch/dirac/shared/opt/proc_lsst/pipelines/submit.yaml -b /mmfs1/home/stevengs/dirac/DEEP/repo -i DEEP/20190504/A0b --output-run DEEP/20190504/A0b/science#step1/20240308T071422Z --qgraph pipeline.qgraph lsst.ctrl.bps.drivers INFO: DISCLAIMER: All values regarding memory consumption reported below are approximate and may not accurately reflect actual memory usage by the bps process. lsst.ctrl.bps.drivers INFO: Starting submission process lsst.ctrl.bps.drivers INFO: Initializing execution environment lsst.ctrl.bps.drivers INFO: Initializing execution environment completed: Took 2.7110 seconds; current memory usage: 0.178 Gibyte, delta: 0.041 Gibyte, peak delta: 0.054 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.190 Gibyte (main), 0.000 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting acquire stage (generating and/or reading quantum graph) lsst.ctrl.bps.pre_transform INFO: Copying quantum graph from 'pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed copying quantum graph: Took 0.0086 seconds lsst.ctrl.bps.pre_transform INFO: Backing up quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed backing up quantum graph: Took 0.0170 seconds lsst.ctrl.bps.pre_transform INFO: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_mpexec/g1ce94f1343+c79f27626b/bin/pipetask --long-log --log-level=VERBOSE update-graph-run /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/pipeline_orig.qgraph DEEP/20190504/A0b/science#step1/20240308T071422Z /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/pipeline.qgraph lsst.ctrl.bps.pre_transform INFO: Reading quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed reading quantum graph: Took 5.6179 seconds lsst.ctrl.bps.drivers INFO: Acquire stage completed: Took 15.9648 seconds; current memory usage: 0.380 Gibyte, delta: 0.202 Gibyte, peak delta: 0.190 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.380 Gibyte (main), 0.370 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting cluster stage (grouping quanta into jobs) lsst.ctrl.bps.drivers INFO: Cluster stage completed: Took 0.2309 seconds; current memory usage: 0.380 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.380 Gibyte (main), 0.370 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: ClusteredQuantumGraph contains 1116 cluster(s) lsst.ctrl.bps.drivers INFO: Starting transform stage (creating generic workflow) lsst.ctrl.bps.drivers INFO: Generic workflow name 'DEEP_20190504_A0b_science#step1_20240308T071422Z' lsst.ctrl.bps.drivers INFO: Transform stage completed: Took 0.2248 seconds; current memory usage: 0.380 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.380 Gibyte (main), 0.370 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: GenericWorkflow contains 1118 job(s) (including final) lsst.ctrl.bps.drivers INFO: Starting prepare stage (creating specific implementation of workflow) parsl.addresses ERROR: Ignoring failure to fetch address from interface eno2 Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 111, in get_all_addresses s_addresses.add(address_by_interface(interface)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py", line 1033, in wrapper retval = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 93, in address_by_interface return socket.inet_ntoa(fcntl.ioctl( ^^^^^^^^^^^^ OSError: [Errno 99] Cannot assign requested address lsst.ctrl.bps.parsl INFO: Writing workflow with ID=/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z lsst.ctrl.bps.drivers INFO: Prepare stage completed: Took 0.3820 seconds; current memory usage: 0.382 Gibyte, delta: 0.002 Gibyte, peak delta: 0.002 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.382 Gibyte (main), 0.370 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting submit stage lsst.ctrl.bps.submit INFO: Submitting run to a workflow management system for execution parsl.dataflow.rundirs DEBUG: Parsl run initializing in rundir: runinfo/000 parsl.dataflow.dflow INFO: Starting DataFlowKernel with config Config( app_cache=True, checkpoint_files=None, checkpoint_mode='task_exit', checkpoint_period=None, executors=[MultiHighThroughputExecutor()], garbage_collect=True, initialize_logging=True, internal_tasks_max_threads=10, max_idletime=120.0, monitoring=None, retries=1, retry_handler=None, run_dir='runinfo', strategy='simple', usage_tracking=False ) parsl.dataflow.dflow INFO: Parsl version: 2023.06.12 parsl.usage_tracking.usage DEBUG: Tracking status: False parsl.dataflow.dflow INFO: Run id is: 29829a25-898d-4c43-b825-cd5509ca6d7b parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_bps_parsl/g145af14111+5b908e21bc/python/lsst/ctrl/bps/parsl/workflow.py parsl.dataflow.dflow DEBUG: Using workflow.py as workflow name parsl.dataflow.memoization INFO: App caching initialized parsl.dataflow.strategy DEBUG: Scaling strategy: simple parsl.executors.high_throughput.executor DEBUG: Starting queue management thread parsl.executors.high_throughput.executor DEBUG: queue management worker starting parsl.executors.high_throughput.executor DEBUG: Started queue management thread Submit dir: /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z parsl.executors.high_throughput.executor DEBUG: Created management thread: parsl.executors.high_throughput.executor DEBUG: Launch command: process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id={block_id} --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.executors.high_throughput.executor DEBUG: Starting HighThroughputExecutor with provider: parsl.executors.status_handling INFO: Scaling out by 1 blocks parsl.executors.status_handling INFO: Allocated block ID 0 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-0 2024-03-08 07:15:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 2024-03-08 07:15:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-08 07:15:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local parsl.providers.local.local DEBUG: Launching in remote mode 2024-03-08 07:15:10 proc_lsst.multi:170 [INFO] [multi] job_id 75775 proc_lsst.multi INFO: [multi] job_id 75775 2024-03-08 07:15:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:10 proc_lsst.multi:178 [INFO] [multi] provider local accepted submit and returned 75775 proc_lsst.multi INFO: [multi] provider local accepted submit and returned 75775 parsl.executors.status_handling DEBUG: Launched block 0 on executor multi with job ID 75775 parsl.dataflow.job_status_poller DEBUG: Adding executor multi parsl.dataflow.dflow DEBUG: Task 0 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 0 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 0 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 0 has memoization hash 1b4244b5a6f29a54d9b8667c52fb483d parsl.dataflow.memoization INFO: Task 0 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467b1c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 0 try 0 launched on executor multi with executor id 1 parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/0e5243e1-505d-4a4e-9768-7671fca81598_isr_855137_15.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/0e5243e1-505d-4a4e-9768-7671fca81598_isr_855137_15.stderr parsl.dataflow.dflow DEBUG: Task 1 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1 submitted for App characterizeImage, waiting on task 0 parsl.dataflow.dflow DEBUG: Task 1 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 2 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2 submitted for App calibrate, waiting on task 1 parsl.dataflow.dflow DEBUG: Task 2 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 2 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 3 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3 has memoization hash 4cdb926e1410e01c1674c0d3ed22b44f parsl.dataflow.memoization INFO: Task 3 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467b2520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3 try 0 launched on executor multi with executor id 2 parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/745c45ee-4ceb-4233-81e7-403a9d11165a_isr_855137_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/745c45ee-4ceb-4233-81e7-403a9d11165a_isr_855137_13.stderr parsl.dataflow.dflow DEBUG: Task 4 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4 submitted for App characterizeImage, waiting on task 3 parsl.dataflow.dflow DEBUG: Task 4 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 4 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 5 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5 submitted for App calibrate, waiting on task 4 parsl.dataflow.dflow DEBUG: Task 5 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 5 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 6 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 6 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 6 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 6 has memoization hash 5cb21205d230da60251cb2a4ac46cf73 parsl.dataflow.memoization INFO: Task 6 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467b27a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 6 try 0 launched on executor multi with executor id 3 parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/7ceaf897-72f4-42d8-9230-00bb5c51213d_isr_855044_41.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/7ceaf897-72f4-42d8-9230-00bb5c51213d_isr_855044_41.stderr parsl.dataflow.dflow DEBUG: Task 7 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 7 submitted for App characterizeImage, waiting on task 6 parsl.dataflow.dflow DEBUG: Task 7 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 7 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 8 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 8 submitted for App calibrate, waiting on task 7 parsl.dataflow.dflow DEBUG: Task 8 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 8 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 9 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 9 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 9 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 9 has memoization hash 7467624ca3ca0e500066db9a25342c3e parsl.dataflow.memoization INFO: Task 9 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467b2980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 9 try 0 launched on executor multi with executor id 4 parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/bfa98fd0-16d3-4f1c-9a2a-9c4ec194570e_isr_855137_61.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/bfa98fd0-16d3-4f1c-9a2a-9c4ec194570e_isr_855137_61.stderr parsl.dataflow.dflow DEBUG: Task 10 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 10 submitted for App characterizeImage, waiting on task 9 parsl.dataflow.dflow DEBUG: Task 10 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 10 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 11 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 11 submitted for App calibrate, waiting on task 10 parsl.dataflow.dflow DEBUG: Task 11 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 11 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 12 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 12 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 12 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 12 has memoization hash 3999b4c4c537a64ae39566a5d078f44a parsl.dataflow.memoization INFO: Task 12 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467b2e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 12 try 0 launched on executor multi with executor id 5 parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/53ae629e-8af0-490e-9ee2-cc3f2fcb179c_isr_855044_4.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/53ae629e-8af0-490e-9ee2-cc3f2fcb179c_isr_855044_4.stderr parsl.dataflow.dflow DEBUG: Task 13 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 13 submitted for App characterizeImage, waiting on task 12 parsl.dataflow.dflow DEBUG: Task 13 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 13 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 14 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 14 submitted for App calibrate, waiting on task 13 parsl.dataflow.dflow DEBUG: Task 14 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 14 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 15 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 15 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 15 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 15 has memoization hash ff4e539b7d6d06fce61bc4a27398f8ce parsl.dataflow.memoization INFO: Task 15 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467b34c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 15 try 0 launched on executor multi with executor id 6 parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/0f1cc0d6-1c97-4cf2-b716-0813e198509d_isr_855044_53.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/0f1cc0d6-1c97-4cf2-b716-0813e198509d_isr_855044_53.stderr parsl.dataflow.dflow DEBUG: Task 16 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 16 submitted for App characterizeImage, waiting on task 15 parsl.dataflow.dflow DEBUG: Task 16 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 16 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 17 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 17 submitted for App calibrate, waiting on task 16 parsl.dataflow.dflow DEBUG: Task 17 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 17 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 18 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 18 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 18 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 18 has memoization hash 573e418b898a50d3ec9a3ebf0fd4fd91 parsl.dataflow.memoization INFO: Task 18 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467b3600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 18 try 0 launched on executor multi with executor id 7 parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/7d053fce-e1a9-44c6-9454-61852b4ad9e7_isr_855044_3.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/7d053fce-e1a9-44c6-9454-61852b4ad9e7_isr_855044_3.stderr parsl.dataflow.dflow DEBUG: Task 19 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 19 submitted for App characterizeImage, waiting on task 18 parsl.dataflow.dflow DEBUG: Task 19 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 19 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 20 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 20 submitted for App calibrate, waiting on task 19 parsl.dataflow.dflow DEBUG: Task 20 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 20 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 21 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 21 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 21 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 21 has memoization hash 4776766eb4a70a59850d944c0c6fbeb5 parsl.dataflow.memoization INFO: Task 21 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467b3880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 21 try 0 launched on executor multi with executor id 8 parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/0b889644-816e-46a9-b3a2-90fa6534baf3_isr_855137_52.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/0b889644-816e-46a9-b3a2-90fa6534baf3_isr_855137_52.stderr parsl.dataflow.dflow DEBUG: Task 22 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 22 submitted for App characterizeImage, waiting on task 21 parsl.dataflow.dflow DEBUG: Task 22 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 22 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 23 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 23 submitted for App calibrate, waiting on task 22 parsl.dataflow.dflow DEBUG: Task 23 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 23 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 24 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 24 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 24 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 24 has memoization hash 01250d59c1f6255c5477d0c604c3b3a2 parsl.dataflow.memoization INFO: Task 24 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467b3d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 24 try 0 launched on executor multi with executor id 9 parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/c1c7848a-4ad2-4eb1-a232-9aaab8993286_isr_855044_22.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/c1c7848a-4ad2-4eb1-a232-9aaab8993286_isr_855044_22.stderr parsl.dataflow.dflow DEBUG: Task 25 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 25 submitted for App characterizeImage, waiting on task 24 parsl.dataflow.dflow DEBUG: Task 25 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 25 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 26 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 26 submitted for App calibrate, waiting on task 25 parsl.dataflow.dflow DEBUG: Task 26 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 26 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 27 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 27 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 27 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 27 has memoization hash d2b46b4b4bf86fabbdfd37636d8a1e38 parsl.dataflow.memoization INFO: Task 27 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467b3a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 27 try 0 launched on executor multi with executor id 10 parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/eb386060-bf2a-4275-ad0b-d9d0624dac5c_isr_855137_19.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/eb386060-bf2a-4275-ad0b-d9d0624dac5c_isr_855137_19.stderr parsl.dataflow.dflow DEBUG: Task 28 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 28 submitted for App characterizeImage, waiting on task 27 parsl.dataflow.dflow DEBUG: Task 28 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 28 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 29 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 29 submitted for App calibrate, waiting on task 28 parsl.dataflow.dflow DEBUG: Task 29 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 29 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 30 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 30 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 30 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 30 has memoization hash 08f98d925f36d8bf810e82d63fcd9c04 parsl.dataflow.memoization INFO: Task 30 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d0220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 30 try 0 launched on executor multi with executor id 11 parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/8c500fdd-f17c-4e5f-818d-307f8e868d29_isr_855043_61.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/8c500fdd-f17c-4e5f-818d-307f8e868d29_isr_855043_61.stderr parsl.dataflow.dflow DEBUG: Task 31 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 31 submitted for App characterizeImage, waiting on task 30 parsl.dataflow.dflow DEBUG: Task 31 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 31 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 32 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 32 submitted for App calibrate, waiting on task 31 parsl.dataflow.dflow DEBUG: Task 32 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 32 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 33 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 33 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 33 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 33 has memoization hash a8105b3ba42e27c05b314eb3fc5ee7be parsl.dataflow.memoization INFO: Task 33 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d0540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 33 try 0 launched on executor multi with executor id 12 parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/5e6c7cb6-c807-4de6-b790-4ed11872c0fd_isr_855137_7.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/5e6c7cb6-c807-4de6-b790-4ed11872c0fd_isr_855137_7.stderr parsl.dataflow.dflow DEBUG: Task 34 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 34 submitted for App characterizeImage, waiting on task 33 parsl.dataflow.dflow DEBUG: Task 34 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 34 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 35 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 35 submitted for App calibrate, waiting on task 34 parsl.dataflow.dflow DEBUG: Task 35 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 35 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 36 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 36 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 36 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 36 has memoization hash 1c58e72f23d919abeccdffed4b00ae68 parsl.dataflow.memoization INFO: Task 36 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d0860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 36 try 0 launched on executor multi with executor id 13 parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/4cdd37f4-e5d4-448e-b0a3-017228c3e3c3_isr_855138_42.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/4cdd37f4-e5d4-448e-b0a3-017228c3e3c3_isr_855138_42.stderr parsl.dataflow.dflow DEBUG: Task 37 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 37 submitted for App characterizeImage, waiting on task 36 parsl.dataflow.dflow DEBUG: Task 37 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 37 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 38 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 38 submitted for App calibrate, waiting on task 37 parsl.dataflow.dflow DEBUG: Task 38 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 38 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 39 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 39 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 39 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 39 has memoization hash 46144cfd084f63f30b44cb0070e17542 parsl.dataflow.memoization INFO: Task 39 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d0c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 39 try 0 launched on executor multi with executor id 14 parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/437c6816-f56c-4593-8bd0-0911f6628b9e_isr_855042_62.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/437c6816-f56c-4593-8bd0-0911f6628b9e_isr_855042_62.stderr parsl.dataflow.dflow DEBUG: Task 40 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 40 submitted for App characterizeImage, waiting on task 39 parsl.dataflow.dflow DEBUG: Task 40 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 40 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 41 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 41 submitted for App calibrate, waiting on task 40 parsl.dataflow.dflow DEBUG: Task 41 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 41 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 42 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 42 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 42 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 42 has memoization hash da67f576481b02e70482a9142c01dd96 parsl.dataflow.memoization INFO: Task 42 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d0fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 42 try 0 launched on executor multi with executor id 15 parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/7675743d-1540-4938-82f0-19a9c8fafe6c_isr_855136_35.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/7675743d-1540-4938-82f0-19a9c8fafe6c_isr_855136_35.stderr parsl.dataflow.dflow DEBUG: Task 43 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 43 submitted for App characterizeImage, waiting on task 42 parsl.dataflow.dflow DEBUG: Task 43 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 43 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 44 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 44 submitted for App calibrate, waiting on task 43 parsl.dataflow.dflow DEBUG: Task 44 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 44 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 45 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 45 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 45 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 45 has memoization hash 6399b24fff620748348edc0e8030088f parsl.dataflow.memoization INFO: Task 45 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d1300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 45 try 0 launched on executor multi with executor id 16 parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/9af661ba-a4a5-4c75-bef9-743e35dc0c79_isr_855138_32.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/9af661ba-a4a5-4c75-bef9-743e35dc0c79_isr_855138_32.stderr parsl.dataflow.dflow DEBUG: Task 46 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 46 submitted for App characterizeImage, waiting on task 45 parsl.dataflow.dflow DEBUG: Task 46 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 46 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 47 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 47 submitted for App calibrate, waiting on task 46 parsl.dataflow.dflow DEBUG: Task 47 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 47 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 48 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 48 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 48 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 48 has memoization hash 631f8aadff31ff50b8c953fefe5e37a6 parsl.dataflow.memoization INFO: Task 48 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d1620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 48 try 0 launched on executor multi with executor id 17 parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/7f4b07d1-b39c-4346-b540-fd4b2b4f3dfb_isr_855138_55.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/7f4b07d1-b39c-4346-b540-fd4b2b4f3dfb_isr_855138_55.stderr parsl.dataflow.dflow DEBUG: Task 49 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 49 submitted for App characterizeImage, waiting on task 48 parsl.dataflow.dflow DEBUG: Task 49 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 49 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 50 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 50 submitted for App calibrate, waiting on task 49 parsl.dataflow.dflow DEBUG: Task 50 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 50 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 51 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 51 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 51 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 51 has memoization hash 233538819bbe48102d9ee02dbe0f6a77 parsl.dataflow.memoization INFO: Task 51 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d1940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 51 try 0 launched on executor multi with executor id 18 parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/3a9c8142-aaed-4475-9d76-e58170699c95_isr_855137_34.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/3a9c8142-aaed-4475-9d76-e58170699c95_isr_855137_34.stderr parsl.dataflow.dflow DEBUG: Task 52 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 52 submitted for App characterizeImage, waiting on task 51 parsl.dataflow.dflow DEBUG: Task 52 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 52 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 53 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 53 submitted for App calibrate, waiting on task 52 parsl.dataflow.dflow DEBUG: Task 53 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 53 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 54 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 54 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 54 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 54 has memoization hash c504d2a58fcc16d018459ddd0b1e3b81 parsl.dataflow.memoization INFO: Task 54 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d1c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 54 try 0 launched on executor multi with executor id 19 parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/6fa0f901-3f62-477c-870c-c34c9e2c6c53_isr_855044_8.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/6fa0f901-3f62-477c-870c-c34c9e2c6c53_isr_855044_8.stderr parsl.dataflow.dflow DEBUG: Task 55 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 55 submitted for App characterizeImage, waiting on task 54 parsl.dataflow.dflow DEBUG: Task 55 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 55 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 56 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 56 submitted for App calibrate, waiting on task 55 parsl.dataflow.dflow DEBUG: Task 56 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 56 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 57 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 57 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 57 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 57 has memoization hash 7b8a977ca53916779e3361d393756b7f parsl.dataflow.memoization INFO: Task 57 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d1f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 57 try 0 launched on executor multi with executor id 20 parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/f33248ce-785c-4bec-93c4-5a8ff29440c1_isr_855136_62.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/f33248ce-785c-4bec-93c4-5a8ff29440c1_isr_855136_62.stderr parsl.dataflow.dflow DEBUG: Task 58 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 58 submitted for App characterizeImage, waiting on task 57 parsl.dataflow.dflow DEBUG: Task 58 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 58 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 59 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 59 submitted for App calibrate, waiting on task 58 parsl.dataflow.dflow DEBUG: Task 59 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 59 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 60 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 60 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 60 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 60 has memoization hash 98560f216fc5432aa4b120676fadd508 parsl.dataflow.memoization INFO: Task 60 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d22a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 60 try 0 launched on executor multi with executor id 21 parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/540a1e13-e110-4fdf-a686-428c42d17156_isr_855138_60.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/540a1e13-e110-4fdf-a686-428c42d17156_isr_855138_60.stderr parsl.dataflow.dflow DEBUG: Task 61 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 61 submitted for App characterizeImage, waiting on task 60 parsl.dataflow.dflow DEBUG: Task 61 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 61 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 62 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 62 submitted for App calibrate, waiting on task 61 parsl.dataflow.dflow DEBUG: Task 62 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 62 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 63 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 63 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 63 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 63 has memoization hash 1c5b81a965d45d625761114c031863d4 parsl.dataflow.memoization INFO: Task 63 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d25c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 63 try 0 launched on executor multi with executor id 22 parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/50ccaf69-3436-4c00-a666-4b7c4ff8004f_isr_855137_31.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/50ccaf69-3436-4c00-a666-4b7c4ff8004f_isr_855137_31.stderr parsl.dataflow.dflow DEBUG: Task 64 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 64 submitted for App characterizeImage, waiting on task 63 parsl.dataflow.dflow DEBUG: Task 64 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 64 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 65 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 65 submitted for App calibrate, waiting on task 64 parsl.dataflow.dflow DEBUG: Task 65 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 65 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 66 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 66 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 66 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 66 has memoization hash cf70f7ea26d1e5de592f964252960bf3 parsl.dataflow.memoization INFO: Task 66 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d28e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 66 try 0 launched on executor multi with executor id 23 parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/5fa47311-04d2-4ef6-8ca3-01de7906989d_isr_855136_34.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/5fa47311-04d2-4ef6-8ca3-01de7906989d_isr_855136_34.stderr parsl.dataflow.dflow DEBUG: Task 67 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 67 submitted for App characterizeImage, waiting on task 66 parsl.dataflow.dflow DEBUG: Task 67 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 67 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 68 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 68 submitted for App calibrate, waiting on task 67 parsl.dataflow.dflow DEBUG: Task 68 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 68 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 69 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 69 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 69 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 69 has memoization hash 84c6d052106aa7324653b86b734a4231 parsl.dataflow.memoization INFO: Task 69 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d2c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 69 try 0 launched on executor multi with executor id 24 parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/8ff78fda-7150-467b-8f20-a32b0bdc346c_isr_855137_4.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/8ff78fda-7150-467b-8f20-a32b0bdc346c_isr_855137_4.stderr parsl.dataflow.dflow DEBUG: Task 70 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 70 submitted for App characterizeImage, waiting on task 69 parsl.dataflow.dflow DEBUG: Task 70 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 70 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 71 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 71 submitted for App calibrate, waiting on task 70 parsl.dataflow.dflow DEBUG: Task 71 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 71 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 72 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 72 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 72 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 72 has memoization hash 7649f3328229f2fc2c987d30439a8ef1 parsl.dataflow.memoization INFO: Task 72 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d2f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 72 try 0 launched on executor multi with executor id 25 parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/253b3c8a-1782-454c-bb6a-7435ec419e0c_isr_855136_13.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/253b3c8a-1782-454c-bb6a-7435ec419e0c_isr_855136_13.stderr parsl.dataflow.dflow DEBUG: Task 73 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 73 submitted for App characterizeImage, waiting on task 72 parsl.dataflow.dflow DEBUG: Task 73 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 73 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 74 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 74 submitted for App calibrate, waiting on task 73 parsl.dataflow.dflow DEBUG: Task 74 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 74 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 75 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 75 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 75 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 75 has memoization hash d9b73ac692db53cb41d2ad2802eca884 parsl.dataflow.memoization INFO: Task 75 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d3240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 75 try 0 launched on executor multi with executor id 26 parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/fcf92283-087a-42f0-bd66-fdc7f7f160be_isr_855136_44.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/fcf92283-087a-42f0-bd66-fdc7f7f160be_isr_855136_44.stderr parsl.dataflow.dflow DEBUG: Task 76 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 76 submitted for App characterizeImage, waiting on task 75 parsl.dataflow.dflow DEBUG: Task 76 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 76 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 77 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 77 submitted for App calibrate, waiting on task 76 parsl.dataflow.dflow DEBUG: Task 77 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 77 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 78 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 78 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 78 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 78 has memoization hash fab42e62c82b9091ecc1a4defcb15d8f parsl.dataflow.memoization INFO: Task 78 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d3560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 78 try 0 launched on executor multi with executor id 27 parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/17e53291-19c2-424a-83b6-f5d90227be32_isr_855136_60.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/17e53291-19c2-424a-83b6-f5d90227be32_isr_855136_60.stderr parsl.dataflow.dflow DEBUG: Task 79 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 79 submitted for App characterizeImage, waiting on task 78 parsl.dataflow.dflow DEBUG: Task 79 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 79 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 80 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 80 submitted for App calibrate, waiting on task 79 parsl.dataflow.dflow DEBUG: Task 80 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 80 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 81 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 81 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 81 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 81 has memoization hash f4ea801dae1c29c6043469a1b4f64de2 parsl.dataflow.memoization INFO: Task 81 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d3880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 81 try 0 launched on executor multi with executor id 28 parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/e6fe17cf-a952-4a1f-81c0-0bae54e15f5b_isr_855138_8.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/e6fe17cf-a952-4a1f-81c0-0bae54e15f5b_isr_855138_8.stderr parsl.dataflow.dflow DEBUG: Task 82 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 82 submitted for App characterizeImage, waiting on task 81 parsl.dataflow.dflow DEBUG: Task 82 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 82 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 83 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 83 submitted for App calibrate, waiting on task 82 parsl.dataflow.dflow DEBUG: Task 83 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 83 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 84 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 84 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 84 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 84 has memoization hash b453e09970d7ebe30fbf33c5675e12df parsl.dataflow.memoization INFO: Task 84 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d3ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 84 try 0 launched on executor multi with executor id 29 parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/3f2772ca-3a2f-4301-a402-8d4fe3c4707f_isr_855137_21.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/3f2772ca-3a2f-4301-a402-8d4fe3c4707f_isr_855137_21.stderr parsl.dataflow.dflow DEBUG: Task 85 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 85 submitted for App characterizeImage, waiting on task 84 parsl.dataflow.dflow DEBUG: Task 85 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 85 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 86 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 86 submitted for App calibrate, waiting on task 85 parsl.dataflow.dflow DEBUG: Task 86 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 86 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 87 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 87 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 87 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 87 has memoization hash b620bc4535a906a8861e3cdc60779310 parsl.dataflow.memoization INFO: Task 87 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d3ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 87 try 0 launched on executor multi with executor id 30 parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/c4155964-541b-41c4-acdc-216ffeec1a46_isr_855044_28.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/c4155964-541b-41c4-acdc-216ffeec1a46_isr_855044_28.stderr parsl.dataflow.dflow DEBUG: Task 88 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 88 submitted for App characterizeImage, waiting on task 87 parsl.dataflow.dflow DEBUG: Task 88 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 88 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 89 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 89 submitted for App calibrate, waiting on task 88 parsl.dataflow.dflow DEBUG: Task 89 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 89 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 90 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 90 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 90 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 90 has memoization hash 0a1c566a1d375d9f83e7a737f60bd1ae parsl.dataflow.memoization INFO: Task 90 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f4220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 90 try 0 launched on executor multi with executor id 31 parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/40113a75-50a3-47d9-a673-a985635251d0_isr_855043_19.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/40113a75-50a3-47d9-a673-a985635251d0_isr_855043_19.stderr parsl.dataflow.dflow DEBUG: Task 91 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 91 submitted for App characterizeImage, waiting on task 90 parsl.dataflow.dflow DEBUG: Task 91 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 91 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 92 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 92 submitted for App calibrate, waiting on task 91 parsl.dataflow.dflow DEBUG: Task 92 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 92 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 93 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 93 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 93 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 93 has memoization hash 5cf6a48f232472395d7640388c1d1d56 parsl.dataflow.memoization INFO: Task 93 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f4540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 93 try 0 launched on executor multi with executor id 32 parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/f0a6b92b-ae86-46bb-bf34-4b350b752a97_isr_855137_30.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/f0a6b92b-ae86-46bb-bf34-4b350b752a97_isr_855137_30.stderr parsl.dataflow.dflow DEBUG: Task 94 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 94 submitted for App characterizeImage, waiting on task 93 parsl.dataflow.dflow DEBUG: Task 94 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 94 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 95 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 95 submitted for App calibrate, waiting on task 94 parsl.dataflow.dflow DEBUG: Task 95 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 95 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 96 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 96 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 96 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 96 has memoization hash 533ff86ca8c7fe6ea042948f3ccb7896 parsl.dataflow.memoization INFO: Task 96 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f4860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 96 try 0 launched on executor multi with executor id 33 parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/1cee0486-f8b4-4aa8-926b-f243c2496b6c_isr_855042_55.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/1cee0486-f8b4-4aa8-926b-f243c2496b6c_isr_855042_55.stderr parsl.dataflow.dflow DEBUG: Task 97 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 97 submitted for App characterizeImage, waiting on task 96 parsl.dataflow.dflow DEBUG: Task 97 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 97 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 98 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 98 submitted for App calibrate, waiting on task 97 parsl.dataflow.dflow DEBUG: Task 98 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 98 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 99 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 99 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 99 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 99 has memoization hash 4bd380644eebb54ff71efff0b723e9f4 parsl.dataflow.memoization INFO: Task 99 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f4b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 99 try 0 launched on executor multi with executor id 34 parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/660d9d5f-a50d-4fb7-b6d1-e15e64f6eae8_isr_855042_49.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/660d9d5f-a50d-4fb7-b6d1-e15e64f6eae8_isr_855042_49.stderr parsl.dataflow.dflow DEBUG: Task 100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 100 submitted for App characterizeImage, waiting on task 99 parsl.dataflow.dflow DEBUG: Task 100 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 100 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 101 submitted for App calibrate, waiting on task 100 parsl.dataflow.dflow DEBUG: Task 101 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 101 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 102 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 102 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 102 has memoization hash 8d45c6969d95b8c59389a8cc01ef1dff parsl.dataflow.memoization INFO: Task 102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f4ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 102 try 0 launched on executor multi with executor id 35 parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/276fb408-af0e-44e2-954c-71b8032f89a2_isr_855136_3.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/276fb408-af0e-44e2-954c-71b8032f89a2_isr_855136_3.stderr parsl.dataflow.dflow DEBUG: Task 103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 103 submitted for App characterizeImage, waiting on task 102 parsl.dataflow.dflow DEBUG: Task 103 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 103 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 104 submitted for App calibrate, waiting on task 103 parsl.dataflow.dflow DEBUG: Task 104 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 104 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 105 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 105 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 105 has memoization hash 0a9ccf97efa854bbed3a7185bc1dfffa parsl.dataflow.memoization INFO: Task 105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f51c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 105 try 0 launched on executor multi with executor id 36 parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/5455d6c8-0ac0-427e-b2a7-daecfb54a4c4_isr_855042_14.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/5455d6c8-0ac0-427e-b2a7-daecfb54a4c4_isr_855042_14.stderr parsl.dataflow.dflow DEBUG: Task 106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 106 submitted for App characterizeImage, waiting on task 105 parsl.dataflow.dflow DEBUG: Task 106 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 106 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 107 submitted for App calibrate, waiting on task 106 parsl.dataflow.dflow DEBUG: Task 107 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 107 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 108 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 108 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 108 has memoization hash 33aa7c855b2184ce111e0bc6d9f964d2 parsl.dataflow.memoization INFO: Task 108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f54e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 108 try 0 launched on executor multi with executor id 37 parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/2609987e-2bba-45ff-8d80-2f785eb100cd_isr_855044_21.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/2609987e-2bba-45ff-8d80-2f785eb100cd_isr_855044_21.stderr parsl.dataflow.dflow DEBUG: Task 109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 109 submitted for App characterizeImage, waiting on task 108 parsl.dataflow.dflow DEBUG: Task 109 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 109 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 110 submitted for App calibrate, waiting on task 109 parsl.dataflow.dflow DEBUG: Task 110 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 110 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 111 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 111 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 111 has memoization hash 08cca887a3a46fe4d5ff40808d734f71 parsl.dataflow.memoization INFO: Task 111 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f5800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 111 try 0 launched on executor multi with executor id 38 parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/1b77263b-0d70-4b1a-b366-2c7005680ffb_isr_855138_21.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/1b77263b-0d70-4b1a-b366-2c7005680ffb_isr_855138_21.stderr parsl.dataflow.dflow DEBUG: Task 112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 112 submitted for App characterizeImage, waiting on task 111 parsl.dataflow.dflow DEBUG: Task 112 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 112 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 113 submitted for App calibrate, waiting on task 112 parsl.dataflow.dflow DEBUG: Task 113 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 113 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 114 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 114 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 114 has memoization hash f7daef65b6711613e9c977e26918aa6a parsl.dataflow.memoization INFO: Task 114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f5b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 114 try 0 launched on executor multi with executor id 39 parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/e4432f35-43f8-4b63-a2f4-869159ee8586_isr_855136_37.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/e4432f35-43f8-4b63-a2f4-869159ee8586_isr_855136_37.stderr parsl.dataflow.dflow DEBUG: Task 115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 115 submitted for App characterizeImage, waiting on task 114 parsl.dataflow.dflow DEBUG: Task 115 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 115 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 116 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 116 submitted for App calibrate, waiting on task 115 parsl.dataflow.dflow DEBUG: Task 116 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 116 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 117 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 117 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 117 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 117 has memoization hash 303da4a94976a4d70ea481d07ee17941 parsl.dataflow.memoization INFO: Task 117 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f5e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 117 try 0 launched on executor multi with executor id 40 parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/15c7ee98-f29e-45b4-9520-12c93565eb34_isr_855042_2.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/15c7ee98-f29e-45b4-9520-12c93565eb34_isr_855042_2.stderr parsl.dataflow.dflow DEBUG: Task 118 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 118 submitted for App characterizeImage, waiting on task 117 parsl.dataflow.dflow DEBUG: Task 118 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 118 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 119 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 119 submitted for App calibrate, waiting on task 118 parsl.dataflow.dflow DEBUG: Task 119 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 119 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 120 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 120 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 120 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 120 has memoization hash cdbd75bed5f97aee9e12698badb72262 parsl.dataflow.memoization INFO: Task 120 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f6160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 120 try 0 launched on executor multi with executor id 41 parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/e65e2958-e469-4f77-a8cb-378dd89514e9_isr_855044_57.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/e65e2958-e469-4f77-a8cb-378dd89514e9_isr_855044_57.stderr parsl.dataflow.dflow DEBUG: Task 121 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 121 submitted for App characterizeImage, waiting on task 120 parsl.dataflow.dflow DEBUG: Task 121 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 121 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 122 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 122 submitted for App calibrate, waiting on task 121 parsl.dataflow.dflow DEBUG: Task 122 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 122 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 123 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 123 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 123 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 123 has memoization hash ab2b5104421e4c0c9031527e628cb842 parsl.dataflow.memoization INFO: Task 123 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f6480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 123 try 0 launched on executor multi with executor id 42 parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/91014b50-bbdf-441b-901b-e93dd03237e9_isr_855137_51.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/91014b50-bbdf-441b-901b-e93dd03237e9_isr_855137_51.stderr parsl.dataflow.dflow DEBUG: Task 124 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 124 submitted for App characterizeImage, waiting on task 123 parsl.dataflow.dflow DEBUG: Task 124 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 124 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 125 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 125 submitted for App calibrate, waiting on task 124 parsl.dataflow.dflow DEBUG: Task 125 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 125 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 126 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 126 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 126 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 126 has memoization hash 1f9b01e1129df6a45c64686aa1ba0086 parsl.dataflow.memoization INFO: Task 126 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f67a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 126 try 0 launched on executor multi with executor id 43 parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/6eec908b-5279-4108-b93c-75d25440488e_isr_855136_42.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/6eec908b-5279-4108-b93c-75d25440488e_isr_855136_42.stderr parsl.dataflow.dflow DEBUG: Task 127 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 127 submitted for App characterizeImage, waiting on task 126 parsl.dataflow.dflow DEBUG: Task 127 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 127 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 128 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 128 submitted for App calibrate, waiting on task 127 parsl.dataflow.dflow DEBUG: Task 128 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 128 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 129 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 129 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 129 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 129 has memoization hash 0bd2b1b23d45065e743d29bcb1130163 parsl.dataflow.memoization INFO: Task 129 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f6ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 129 try 0 launched on executor multi with executor id 44 parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/39c656c9-b606-45cb-8172-24f3cd5f4289_isr_855042_3.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/39c656c9-b606-45cb-8172-24f3cd5f4289_isr_855042_3.stderr parsl.dataflow.dflow DEBUG: Task 130 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 130 submitted for App characterizeImage, waiting on task 129 parsl.dataflow.dflow DEBUG: Task 130 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 130 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 131 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 131 submitted for App calibrate, waiting on task 130 parsl.dataflow.dflow DEBUG: Task 131 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 131 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 132 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 132 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 132 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 132 has memoization hash 897ad8657a6476ede461994d2cc40277 parsl.dataflow.memoization INFO: Task 132 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f6de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 132 try 0 launched on executor multi with executor id 45 parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/cd2f75e7-2322-4a73-af55-fd6ad666dd2a_isr_855044_45.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/cd2f75e7-2322-4a73-af55-fd6ad666dd2a_isr_855044_45.stderr parsl.dataflow.dflow DEBUG: Task 133 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 133 submitted for App characterizeImage, waiting on task 132 parsl.dataflow.dflow DEBUG: Task 133 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 133 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 134 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 134 submitted for App calibrate, waiting on task 133 parsl.dataflow.dflow DEBUG: Task 134 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 134 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 135 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 135 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 135 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 135 has memoization hash 9ebcee17d6f90a2aa2c23ae3e4e321d7 parsl.dataflow.memoization INFO: Task 135 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f7100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 135 try 0 launched on executor multi with executor id 46 parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/2ca50aa6-1a26-4a54-b2f0-232bbcc7364c_isr_855137_57.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/2ca50aa6-1a26-4a54-b2f0-232bbcc7364c_isr_855137_57.stderr parsl.dataflow.dflow DEBUG: Task 136 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 136 submitted for App characterizeImage, waiting on task 135 parsl.dataflow.dflow DEBUG: Task 136 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 136 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 137 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 137 submitted for App calibrate, waiting on task 136 parsl.dataflow.dflow DEBUG: Task 137 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 137 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 138 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 138 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 138 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 138 has memoization hash 02ac701d3db3800fe8d9ac011a6570b7 parsl.dataflow.memoization INFO: Task 138 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f7420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 138 try 0 launched on executor multi with executor id 47 parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/f0fbb9f6-d361-495b-95b6-2f50813275e4_isr_855044_34.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/f0fbb9f6-d361-495b-95b6-2f50813275e4_isr_855044_34.stderr parsl.dataflow.dflow DEBUG: Task 139 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 139 submitted for App characterizeImage, waiting on task 138 parsl.dataflow.dflow DEBUG: Task 139 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 139 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 140 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 140 submitted for App calibrate, waiting on task 139 parsl.dataflow.dflow DEBUG: Task 140 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 140 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 141 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 141 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 141 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 141 has memoization hash 1c17fb59da70de0c91f96a8b135084d2 parsl.dataflow.memoization INFO: Task 141 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f7740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 141 try 0 launched on executor multi with executor id 48 parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/88e3bc28-9b06-4833-b52d-802f02ca873d_isr_855044_24.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/88e3bc28-9b06-4833-b52d-802f02ca873d_isr_855044_24.stderr parsl.dataflow.dflow DEBUG: Task 142 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 142 submitted for App characterizeImage, waiting on task 141 parsl.dataflow.dflow DEBUG: Task 142 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 142 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 143 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 143 submitted for App calibrate, waiting on task 142 parsl.dataflow.dflow DEBUG: Task 143 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 143 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 144 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 144 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 144 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 144 has memoization hash 6d5bcc7b86757a903ac35a91f786ade6 parsl.dataflow.memoization INFO: Task 144 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f7a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 144 try 0 launched on executor multi with executor id 49 parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/c8ca8d8e-01e3-48c2-9e5e-2eeff2e48ab6_isr_855042_45.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/c8ca8d8e-01e3-48c2-9e5e-2eeff2e48ab6_isr_855042_45.stderr parsl.dataflow.dflow DEBUG: Task 145 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 145 submitted for App characterizeImage, waiting on task 144 parsl.dataflow.dflow DEBUG: Task 145 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 145 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 146 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 146 submitted for App calibrate, waiting on task 145 parsl.dataflow.dflow DEBUG: Task 146 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 146 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 147 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 147 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 147 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 147 has memoization hash 4120ada7ea71a5f7a806145ce720dabb parsl.dataflow.memoization INFO: Task 147 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f7d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 147 try 0 launched on executor multi with executor id 50 parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/b82f5527-1e2e-4403-a810-9fb23276c8bd_isr_855138_37.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/b82f5527-1e2e-4403-a810-9fb23276c8bd_isr_855138_37.stderr parsl.dataflow.dflow DEBUG: Task 148 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 148 submitted for App characterizeImage, waiting on task 147 parsl.dataflow.dflow DEBUG: Task 148 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 148 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 149 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 149 submitted for App calibrate, waiting on task 148 parsl.dataflow.dflow DEBUG: Task 149 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 149 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 150 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 150 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 150 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 150 has memoization hash 9ad971c09e4c47be235d056266b249f6 parsl.dataflow.memoization INFO: Task 150 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa468140e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 150 try 0 launched on executor multi with executor id 51 parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/d4729193-6884-40e6-bf93-7334472f419e_isr_855043_29.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/d4729193-6884-40e6-bf93-7334472f419e_isr_855043_29.stderr parsl.dataflow.dflow DEBUG: Task 151 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 151 submitted for App characterizeImage, waiting on task 150 parsl.dataflow.dflow DEBUG: Task 151 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 151 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 152 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 152 submitted for App calibrate, waiting on task 151 parsl.dataflow.dflow DEBUG: Task 152 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 152 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 153 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 153 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 153 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 153 has memoization hash 3cb700137e4d49b8cd17a89cabd1e9a3 parsl.dataflow.memoization INFO: Task 153 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46814400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 153 try 0 launched on executor multi with executor id 52 parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/7ee269cb-5ebb-4149-b13a-7f9461be3e9e_isr_855138_56.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/7ee269cb-5ebb-4149-b13a-7f9461be3e9e_isr_855138_56.stderr parsl.dataflow.dflow DEBUG: Task 154 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 154 submitted for App characterizeImage, waiting on task 153 parsl.dataflow.dflow DEBUG: Task 154 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 154 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 155 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 155 submitted for App calibrate, waiting on task 154 parsl.dataflow.dflow DEBUG: Task 155 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 155 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 156 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 156 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 156 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 156 has memoization hash 8197dafb71467eb7bfbe9c0fb0f75da8 parsl.dataflow.memoization INFO: Task 156 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46814720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 156 try 0 launched on executor multi with executor id 53 parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/65becc69-abba-4e78-9acf-e8166c3c152d_isr_855043_55.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/65becc69-abba-4e78-9acf-e8166c3c152d_isr_855043_55.stderr parsl.dataflow.dflow DEBUG: Task 157 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 157 submitted for App characterizeImage, waiting on task 156 parsl.dataflow.dflow DEBUG: Task 157 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 157 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 158 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 158 submitted for App calibrate, waiting on task 157 parsl.dataflow.dflow DEBUG: Task 158 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 158 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 159 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 159 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 159 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 159 has memoization hash d7dae35413fd80a74962643f5376fe88 parsl.dataflow.memoization INFO: Task 159 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46814a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 159 try 0 launched on executor multi with executor id 54 parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/0d24145e-1ed2-42b0-880a-405e6601838b_isr_855138_57.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/0d24145e-1ed2-42b0-880a-405e6601838b_isr_855138_57.stderr parsl.dataflow.dflow DEBUG: Task 160 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 160 submitted for App characterizeImage, waiting on task 159 parsl.dataflow.dflow DEBUG: Task 160 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 160 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 161 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 161 submitted for App calibrate, waiting on task 160 parsl.dataflow.dflow DEBUG: Task 161 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 161 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 162 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 162 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 162 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 162 has memoization hash bd4d7ac0138224c419ac4863a4b16224 parsl.dataflow.memoization INFO: Task 162 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46814d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 162 try 0 launched on executor multi with executor id 55 parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/dddd104b-3799-4bb2-8d3b-6118a5f03349_isr_855042_35.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/dddd104b-3799-4bb2-8d3b-6118a5f03349_isr_855042_35.stderr parsl.dataflow.dflow DEBUG: Task 163 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 163 submitted for App characterizeImage, waiting on task 162 parsl.dataflow.dflow DEBUG: Task 163 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 163 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 164 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 164 submitted for App calibrate, waiting on task 163 parsl.dataflow.dflow DEBUG: Task 164 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 164 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 165 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 165 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 165 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 165 has memoization hash 6bbb0abf54ab7558b8100bafa9fcf921 parsl.dataflow.memoization INFO: Task 165 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46815080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 165 try 0 launched on executor multi with executor id 56 parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/898687d0-0e30-4ee7-b8a9-f5cab23999b5_isr_855042_20.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/898687d0-0e30-4ee7-b8a9-f5cab23999b5_isr_855042_20.stderr parsl.dataflow.dflow DEBUG: Task 166 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 166 submitted for App characterizeImage, waiting on task 165 parsl.dataflow.dflow DEBUG: Task 166 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 166 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 167 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 167 submitted for App calibrate, waiting on task 166 parsl.dataflow.dflow DEBUG: Task 167 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 167 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 168 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 168 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 168 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 168 has memoization hash 126fe71ff91b0402f19333144fe46f28 parsl.dataflow.memoization INFO: Task 168 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa468153a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 168 try 0 launched on executor multi with executor id 57 parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/3823d9b7-ef0e-4a4e-89cf-aad04a6be695_isr_855042_39.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/3823d9b7-ef0e-4a4e-89cf-aad04a6be695_isr_855042_39.stderr parsl.dataflow.dflow DEBUG: Task 169 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 169 submitted for App characterizeImage, waiting on task 168 parsl.dataflow.dflow DEBUG: Task 169 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 169 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 170 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 170 submitted for App calibrate, waiting on task 169 parsl.dataflow.dflow DEBUG: Task 170 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 170 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 171 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 171 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 171 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 171 has memoization hash 0361ab0ed6fe33abe77aac5b2140dbe2 parsl.dataflow.memoization INFO: Task 171 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa468156c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 171 try 0 launched on executor multi with executor id 58 parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/ffb275d3-f855-4378-93c8-22bb9eef0d15_isr_855138_49.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/ffb275d3-f855-4378-93c8-22bb9eef0d15_isr_855138_49.stderr parsl.dataflow.dflow DEBUG: Task 172 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 172 submitted for App characterizeImage, waiting on task 171 parsl.dataflow.dflow DEBUG: Task 172 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 172 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 173 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 173 submitted for App calibrate, waiting on task 172 parsl.dataflow.dflow DEBUG: Task 173 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 173 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 174 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 174 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 174 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 174 has memoization hash f2a468bf8c587908fea839c4bb4278cf parsl.dataflow.memoization INFO: Task 174 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa468159e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 174 try 0 launched on executor multi with executor id 59 parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/90f8511f-e07a-41ac-9b26-4a1dd8a91393_isr_855043_59.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/90f8511f-e07a-41ac-9b26-4a1dd8a91393_isr_855043_59.stderr parsl.dataflow.dflow DEBUG: Task 175 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 175 submitted for App characterizeImage, waiting on task 174 parsl.dataflow.dflow DEBUG: Task 175 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 175 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 176 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 176 submitted for App calibrate, waiting on task 175 parsl.dataflow.dflow DEBUG: Task 176 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 176 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 177 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 177 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 177 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 177 has memoization hash 5733044278bc75a02b99e2d91e9d8f8a parsl.dataflow.memoization INFO: Task 177 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46815d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 177 try 0 launched on executor multi with executor id 60 parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/3800fef7-f63b-4327-98e3-3ba1cc2637d3_isr_855137_62.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/3800fef7-f63b-4327-98e3-3ba1cc2637d3_isr_855137_62.stderr parsl.dataflow.dflow DEBUG: Task 178 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 178 submitted for App characterizeImage, waiting on task 177 parsl.dataflow.dflow DEBUG: Task 178 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 178 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 179 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 179 submitted for App calibrate, waiting on task 178 parsl.dataflow.dflow DEBUG: Task 179 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 179 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 180 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 180 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 180 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 180 has memoization hash 72697841cfc64beb4af3d64be7bac29b parsl.dataflow.memoization INFO: Task 180 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46816020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 180 try 0 launched on executor multi with executor id 61 parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/4ff7ad7f-c353-4ec9-a506-c0cb052d32eb_isr_855044_58.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/4ff7ad7f-c353-4ec9-a506-c0cb052d32eb_isr_855044_58.stderr parsl.dataflow.dflow DEBUG: Task 181 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 181 submitted for App characterizeImage, waiting on task 180 parsl.dataflow.dflow DEBUG: Task 181 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 181 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 182 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 182 submitted for App calibrate, waiting on task 181 parsl.dataflow.dflow DEBUG: Task 182 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 182 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 183 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 183 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 183 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 183 has memoization hash ca49b363bcee2def3c4814c7f32c8cfa parsl.dataflow.memoization INFO: Task 183 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46816340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 183 try 0 launched on executor multi with executor id 62 parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/6c5bf872-099f-40ce-a235-0b23b31e0abd_isr_855137_6.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/6c5bf872-099f-40ce-a235-0b23b31e0abd_isr_855137_6.stderr parsl.dataflow.dflow DEBUG: Task 184 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 184 submitted for App characterizeImage, waiting on task 183 parsl.dataflow.dflow DEBUG: Task 184 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 184 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 185 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 185 submitted for App calibrate, waiting on task 184 parsl.dataflow.dflow DEBUG: Task 185 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 185 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 186 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 186 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 186 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 186 has memoization hash ad775c156c6bad418000312f3adc78fb parsl.dataflow.memoization INFO: Task 186 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46816660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 186 try 0 launched on executor multi with executor id 63 parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/5b1028ef-b906-4f28-a08a-4d4859621ffb_isr_855044_44.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/5b1028ef-b906-4f28-a08a-4d4859621ffb_isr_855044_44.stderr parsl.dataflow.dflow DEBUG: Task 187 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 187 submitted for App characterizeImage, waiting on task 186 parsl.dataflow.dflow DEBUG: Task 187 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 187 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 188 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 188 submitted for App calibrate, waiting on task 187 parsl.dataflow.dflow DEBUG: Task 188 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 188 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 189 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 189 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 189 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 189 has memoization hash e8171df3ebe51bff72a33b32724dce09 parsl.dataflow.memoization INFO: Task 189 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46816980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 189 try 0 launched on executor multi with executor id 64 parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/36c871f6-0028-4c71-8df5-daa791642dbd_isr_855044_60.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/36c871f6-0028-4c71-8df5-daa791642dbd_isr_855044_60.stderr parsl.dataflow.dflow DEBUG: Task 190 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 190 submitted for App characterizeImage, waiting on task 189 parsl.dataflow.dflow DEBUG: Task 190 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 190 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 191 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 191 submitted for App calibrate, waiting on task 190 parsl.dataflow.dflow DEBUG: Task 191 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 191 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 192 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 192 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 192 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 192 has memoization hash ba6a1d805b868573074901cc4638f9b4 parsl.dataflow.memoization INFO: Task 192 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46816ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 192 try 0 launched on executor multi with executor id 65 parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/73856558-3994-454b-869e-10ae4e73aedf_isr_855136_33.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/73856558-3994-454b-869e-10ae4e73aedf_isr_855136_33.stderr parsl.dataflow.dflow DEBUG: Task 193 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 193 submitted for App characterizeImage, waiting on task 192 parsl.dataflow.dflow DEBUG: Task 193 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 193 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 194 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 194 submitted for App calibrate, waiting on task 193 parsl.dataflow.dflow DEBUG: Task 194 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 194 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 195 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 195 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 195 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 195 has memoization hash ac2a79298b41a749d3c9922fe4687178 parsl.dataflow.memoization INFO: Task 195 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46816fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 195 try 0 launched on executor multi with executor id 66 parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/4e5967a7-3e73-45a6-9974-76bed61e140f_isr_855136_45.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/4e5967a7-3e73-45a6-9974-76bed61e140f_isr_855136_45.stderr parsl.dataflow.dflow DEBUG: Task 196 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 196 submitted for App characterizeImage, waiting on task 195 parsl.dataflow.dflow DEBUG: Task 196 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 196 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 197 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 197 submitted for App calibrate, waiting on task 196 parsl.dataflow.dflow DEBUG: Task 197 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 197 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 198 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 198 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 198 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 198 has memoization hash a10d0f3b9d07c72750bf5a589fc16434 parsl.dataflow.memoization INFO: Task 198 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa468172e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 198 try 0 launched on executor multi with executor id 67 parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/71ed56e4-60aa-4d3d-9eb9-4435c1077753_isr_855042_10.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/71ed56e4-60aa-4d3d-9eb9-4435c1077753_isr_855042_10.stderr parsl.dataflow.dflow DEBUG: Task 199 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 199 submitted for App characterizeImage, waiting on task 198 parsl.dataflow.dflow DEBUG: Task 199 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 199 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 200 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 200 submitted for App calibrate, waiting on task 199 parsl.dataflow.dflow DEBUG: Task 200 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 200 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 201 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 201 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 201 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 201 has memoization hash d25fb8e613b464395dcc51c808315446 parsl.dataflow.memoization INFO: Task 201 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46817600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 201 try 0 launched on executor multi with executor id 68 parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/fd75a794-b0b4-487b-830d-2e75bcbef545_isr_855138_33.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/fd75a794-b0b4-487b-830d-2e75bcbef545_isr_855138_33.stderr parsl.dataflow.dflow DEBUG: Task 202 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 202 submitted for App characterizeImage, waiting on task 201 parsl.dataflow.dflow DEBUG: Task 202 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 202 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 203 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 203 submitted for App calibrate, waiting on task 202 parsl.dataflow.dflow DEBUG: Task 203 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 203 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 204 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 204 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 204 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 204 has memoization hash 9c2fb1676b6c266a99dbd4a574af6d46 parsl.dataflow.memoization INFO: Task 204 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46817920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 204 try 0 launched on executor multi with executor id 69 parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/862d6136-bf44-46a8-8c2e-364e100f2401_isr_855044_33.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/862d6136-bf44-46a8-8c2e-364e100f2401_isr_855044_33.stderr parsl.dataflow.dflow DEBUG: Task 205 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 205 submitted for App characterizeImage, waiting on task 204 parsl.dataflow.dflow DEBUG: Task 205 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 205 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 206 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 206 submitted for App calibrate, waiting on task 205 parsl.dataflow.dflow DEBUG: Task 206 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 206 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 207 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 207 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 207 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 207 has memoization hash dde581b5fe85abb1ac32305bb8a85fa7 parsl.dataflow.memoization INFO: Task 207 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46817c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 207 try 0 launched on executor multi with executor id 70 parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/19394193-1a40-4edf-bc70-5a882a580a65_isr_855042_17.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/19394193-1a40-4edf-bc70-5a882a580a65_isr_855042_17.stderr parsl.dataflow.dflow DEBUG: Task 208 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 208 submitted for App characterizeImage, waiting on task 207 parsl.dataflow.dflow DEBUG: Task 208 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 208 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 209 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 209 submitted for App calibrate, waiting on task 208 parsl.dataflow.dflow DEBUG: Task 209 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 209 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 210 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 210 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 210 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 210 has memoization hash e70ec555dd4406c4854ab4611e51831f parsl.dataflow.memoization INFO: Task 210 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46817f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 210 try 0 launched on executor multi with executor id 71 parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/a117a6fa-232a-4b19-8fce-18f3cef48883_isr_855137_33.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/a117a6fa-232a-4b19-8fce-18f3cef48883_isr_855137_33.stderr parsl.dataflow.dflow DEBUG: Task 211 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 211 submitted for App characterizeImage, waiting on task 210 parsl.dataflow.dflow DEBUG: Task 211 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 211 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 212 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 212 submitted for App calibrate, waiting on task 211 parsl.dataflow.dflow DEBUG: Task 212 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 212 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 213 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 213 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 213 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 213 has memoization hash 85aeab72fe85f302cd73dd1eff616393 parsl.dataflow.memoization INFO: Task 213 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa468342c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 213 try 0 launched on executor multi with executor id 72 parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/157946d2-b278-4387-8aa5-e25c2c43dd37_isr_855043_12.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/157946d2-b278-4387-8aa5-e25c2c43dd37_isr_855043_12.stderr parsl.dataflow.dflow DEBUG: Task 214 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 214 submitted for App characterizeImage, waiting on task 213 parsl.dataflow.dflow DEBUG: Task 214 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 214 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 215 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 215 submitted for App calibrate, waiting on task 214 parsl.dataflow.dflow DEBUG: Task 215 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 215 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 216 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 216 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 216 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 216 has memoization hash 20ade8268ccf15a1ae8ea0f7835e0b3e parsl.dataflow.memoization INFO: Task 216 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa468345e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 216 try 0 launched on executor multi with executor id 73 parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/e001ba52-f609-4c5e-877e-5e0ab69d7658_isr_855043_17.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/e001ba52-f609-4c5e-877e-5e0ab69d7658_isr_855043_17.stderr parsl.dataflow.dflow DEBUG: Task 217 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 217 submitted for App characterizeImage, waiting on task 216 parsl.dataflow.dflow DEBUG: Task 217 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 217 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 218 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 218 submitted for App calibrate, waiting on task 217 parsl.dataflow.dflow DEBUG: Task 218 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 218 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 219 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 219 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 219 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 219 has memoization hash 3bdd8e02cbbcdef044f0db3e0752ea1b parsl.dataflow.memoization INFO: Task 219 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46834900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 219 try 0 launched on executor multi with executor id 74 parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/b37a1860-7784-41e7-9b87-336efae3ea1d_isr_855043_32.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/b37a1860-7784-41e7-9b87-336efae3ea1d_isr_855043_32.stderr parsl.dataflow.dflow DEBUG: Task 220 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 220 submitted for App characterizeImage, waiting on task 219 parsl.dataflow.dflow DEBUG: Task 220 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 220 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 221 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 221 submitted for App calibrate, waiting on task 220 parsl.dataflow.dflow DEBUG: Task 221 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 221 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 222 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 222 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 222 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 222 has memoization hash dae6ce8fd412b65cb5151f1a76bed5eb parsl.dataflow.memoization INFO: Task 222 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46834c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 222 try 0 launched on executor multi with executor id 75 parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/9dde3eb1-1348-411c-8d60-34eff42df87e_isr_855044_32.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/9dde3eb1-1348-411c-8d60-34eff42df87e_isr_855044_32.stderr parsl.dataflow.dflow DEBUG: Task 223 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 223 submitted for App characterizeImage, waiting on task 222 parsl.dataflow.dflow DEBUG: Task 223 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 223 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 224 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 224 submitted for App calibrate, waiting on task 223 parsl.dataflow.dflow DEBUG: Task 224 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 224 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 225 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 225 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 225 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 225 has memoization hash 726bf038baa5a597c8cae355fac24390 parsl.dataflow.memoization INFO: Task 225 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46834f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 225 try 0 launched on executor multi with executor id 76 parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/04f21561-c52e-44f8-8e04-5bdd236579d2_isr_855137_48.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/04f21561-c52e-44f8-8e04-5bdd236579d2_isr_855137_48.stderr parsl.dataflow.dflow DEBUG: Task 226 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 226 submitted for App characterizeImage, waiting on task 225 parsl.dataflow.dflow DEBUG: Task 226 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 226 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 227 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 227 submitted for App calibrate, waiting on task 226 parsl.dataflow.dflow DEBUG: Task 227 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 227 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 228 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 228 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 228 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 228 has memoization hash 88ca0a9a10653fa6d8a14c1fc6241ba4 parsl.dataflow.memoization INFO: Task 228 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46835260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 228 try 0 launched on executor multi with executor id 77 parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/9fc95394-35ff-49aa-84ba-359555b5041c_isr_855137_32.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/9fc95394-35ff-49aa-84ba-359555b5041c_isr_855137_32.stderr parsl.dataflow.dflow DEBUG: Task 229 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 229 submitted for App characterizeImage, waiting on task 228 parsl.dataflow.dflow DEBUG: Task 229 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 229 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 230 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 230 submitted for App calibrate, waiting on task 229 parsl.dataflow.dflow DEBUG: Task 230 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 230 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 231 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 231 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 231 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 231 has memoization hash 46bb259300424cb595dde3417412442e parsl.dataflow.memoization INFO: Task 231 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46835580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 231 try 0 launched on executor multi with executor id 78 parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/f8863c15-6528-4445-ad21-17fd41d9172d_isr_855044_48.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/f8863c15-6528-4445-ad21-17fd41d9172d_isr_855044_48.stderr parsl.dataflow.dflow DEBUG: Task 232 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 232 submitted for App characterizeImage, waiting on task 231 parsl.dataflow.dflow DEBUG: Task 232 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 232 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 233 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 233 submitted for App calibrate, waiting on task 232 parsl.dataflow.dflow DEBUG: Task 233 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 233 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 234 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 234 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 234 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 234 has memoization hash d55668be85cad1e63ac05fce0f722721 parsl.dataflow.memoization INFO: Task 234 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa468358a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 234 try 0 launched on executor multi with executor id 79 parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/68822afe-821e-4019-9439-1722d5ccb8bf_isr_855044_16.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/68822afe-821e-4019-9439-1722d5ccb8bf_isr_855044_16.stderr parsl.dataflow.dflow DEBUG: Task 235 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 235 submitted for App characterizeImage, waiting on task 234 parsl.dataflow.dflow DEBUG: Task 235 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 235 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 236 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 236 submitted for App calibrate, waiting on task 235 parsl.dataflow.dflow DEBUG: Task 236 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 236 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 237 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 237 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 237 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 237 has memoization hash 7742b34bdf789918ae6d185cd8c0a2e3 parsl.dataflow.memoization INFO: Task 237 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46835bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 237 try 0 launched on executor multi with executor id 80 parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/dc49aeee-2306-4781-8d09-9c754c1e4564_isr_855138_4.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/dc49aeee-2306-4781-8d09-9c754c1e4564_isr_855138_4.stderr parsl.dataflow.dflow DEBUG: Task 238 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 238 submitted for App characterizeImage, waiting on task 237 parsl.dataflow.dflow DEBUG: Task 238 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 238 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 239 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 239 submitted for App calibrate, waiting on task 238 parsl.dataflow.dflow DEBUG: Task 239 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 239 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 240 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 240 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 240 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 240 has memoization hash a8f1807c6ea7cfeaeb89e751173c3320 parsl.dataflow.memoization INFO: Task 240 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46835ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 240 try 0 launched on executor multi with executor id 81 parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/0ab5ff96-5836-43a9-a6b4-6f7e0b17945a_isr_855136_8.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/0ab5ff96-5836-43a9-a6b4-6f7e0b17945a_isr_855136_8.stderr parsl.dataflow.dflow DEBUG: Task 241 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 241 submitted for App characterizeImage, waiting on task 240 parsl.dataflow.dflow DEBUG: Task 241 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 241 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 242 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 242 submitted for App calibrate, waiting on task 241 parsl.dataflow.dflow DEBUG: Task 242 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 242 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 243 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 243 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 243 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 243 has memoization hash 151dea84198ad2a7eb38407dfe528f30 parsl.dataflow.memoization INFO: Task 243 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46836200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 243 try 0 launched on executor multi with executor id 82 parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/1e777f90-1c49-4e07-b97a-6bc34ef90168_isr_855043_13.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/1e777f90-1c49-4e07-b97a-6bc34ef90168_isr_855043_13.stderr parsl.dataflow.dflow DEBUG: Task 244 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 244 submitted for App characterizeImage, waiting on task 243 parsl.dataflow.dflow DEBUG: Task 244 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 244 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 245 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 245 submitted for App calibrate, waiting on task 244 parsl.dataflow.dflow DEBUG: Task 245 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 245 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 246 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 246 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 246 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 246 has memoization hash 77767580f0d7222b28bb1bcb6cc3bd6e parsl.dataflow.memoization INFO: Task 246 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46836520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 246 try 0 launched on executor multi with executor id 83 parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/79041e03-8b91-4975-a865-0fb8815f55e2_isr_855136_48.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/79041e03-8b91-4975-a865-0fb8815f55e2_isr_855136_48.stderr parsl.dataflow.dflow DEBUG: Task 247 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 247 submitted for App characterizeImage, waiting on task 246 parsl.dataflow.dflow DEBUG: Task 247 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 247 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 248 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 248 submitted for App calibrate, waiting on task 247 parsl.dataflow.dflow DEBUG: Task 248 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 248 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 249 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 249 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 249 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 249 has memoization hash 7a868213f229530899fd644027ae8d18 parsl.dataflow.memoization INFO: Task 249 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46836840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 249 try 0 launched on executor multi with executor id 84 parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/7aaa773e-1a01-4d55-9c56-be3780ab46e4_isr_855136_36.stdout parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/7aaa773e-1a01-4d55-9c56-be3780ab46e4_isr_855136_36.stderr parsl.dataflow.dflow DEBUG: Task 250 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 250 submitted for App characterizeImage, waiting on task 249 parsl.dataflow.dflow DEBUG: Task 250 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 250 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 251 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 251 submitted for App calibrate, waiting on task 250 parsl.dataflow.dflow DEBUG: Task 251 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 251 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 252 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 252 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 252 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 252 has memoization hash ec3caefe65632c85bfcc4198ed93234b parsl.dataflow.memoization INFO: Task 252 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46836b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 252 try 0 launched on executor multi with executor id 85 parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/31469ecf-567f-4283-933a-d2eb85cdc589_isr_855043_6.stdout parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/31469ecf-567f-4283-933a-d2eb85cdc589_isr_855043_6.stderr parsl.dataflow.dflow DEBUG: Task 253 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 253 submitted for App characterizeImage, waiting on task 252 parsl.dataflow.dflow DEBUG: Task 253 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 253 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 254 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 254 submitted for App calibrate, waiting on task 253 parsl.dataflow.dflow DEBUG: Task 254 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 254 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 255 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 255 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 255 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 255 has memoization hash 5763f630926476536e0d274d6008364e parsl.dataflow.memoization INFO: Task 255 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46836e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 255 try 0 launched on executor multi with executor id 86 parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/1402e7ec-156e-46fd-8ed0-4b51f52891c3_isr_855042_43.stdout parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/1402e7ec-156e-46fd-8ed0-4b51f52891c3_isr_855042_43.stderr parsl.dataflow.dflow DEBUG: Task 256 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 256 submitted for App characterizeImage, waiting on task 255 parsl.dataflow.dflow DEBUG: Task 256 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 256 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 257 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 257 submitted for App calibrate, waiting on task 256 parsl.dataflow.dflow DEBUG: Task 257 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 257 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 258 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 258 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 258 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 258 has memoization hash 57be83cb43c87bccc220cbe13a22b5bc parsl.dataflow.memoization INFO: Task 258 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa468371a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 258 try 0 launched on executor multi with executor id 87 parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/f1c58a52-16b0-454e-8310-0f11af24cf04_isr_855137_5.stdout parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/f1c58a52-16b0-454e-8310-0f11af24cf04_isr_855137_5.stderr parsl.dataflow.dflow DEBUG: Task 259 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 259 submitted for App characterizeImage, waiting on task 258 parsl.dataflow.dflow DEBUG: Task 259 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 259 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 260 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 260 submitted for App calibrate, waiting on task 259 parsl.dataflow.dflow DEBUG: Task 260 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 260 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 261 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 261 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 261 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 261 has memoization hash be54af2b403d98943c2d5f53a3bfb9c0 parsl.dataflow.memoization INFO: Task 261 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa468374c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 261 try 0 launched on executor multi with executor id 88 parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/4b8613b9-dd8a-4914-af5a-a089809ab8fd_isr_855136_21.stdout parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/4b8613b9-dd8a-4914-af5a-a089809ab8fd_isr_855136_21.stderr parsl.dataflow.dflow DEBUG: Task 262 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 262 submitted for App characterizeImage, waiting on task 261 parsl.dataflow.dflow DEBUG: Task 262 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 262 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 263 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 263 submitted for App calibrate, waiting on task 262 parsl.dataflow.dflow DEBUG: Task 263 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 263 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 264 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 264 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 264 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 264 has memoization hash 24da08f7f2eb10fa17a82a08037b36f9 parsl.dataflow.memoization INFO: Task 264 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa468377e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 264 try 0 launched on executor multi with executor id 89 parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/0e8f3fbe-0214-4d49-9a6a-9ac27a09f774_isr_855044_40.stdout parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/0e8f3fbe-0214-4d49-9a6a-9ac27a09f774_isr_855044_40.stderr parsl.dataflow.dflow DEBUG: Task 265 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 265 submitted for App characterizeImage, waiting on task 264 parsl.dataflow.dflow DEBUG: Task 265 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 265 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 266 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 266 submitted for App calibrate, waiting on task 265 parsl.dataflow.dflow DEBUG: Task 266 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 266 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 267 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 267 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 267 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 267 has memoization hash fc6ab4e55647b637cc37cb88bd0d8d56 parsl.dataflow.memoization INFO: Task 267 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46837b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 267 try 0 launched on executor multi with executor id 90 parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/9427fa18-fe7a-4764-bd02-53fff1225684_isr_855136_14.stdout parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/9427fa18-fe7a-4764-bd02-53fff1225684_isr_855136_14.stderr parsl.dataflow.dflow DEBUG: Task 268 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 268 submitted for App characterizeImage, waiting on task 267 parsl.dataflow.dflow DEBUG: Task 268 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 268 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 269 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 269 submitted for App calibrate, waiting on task 268 parsl.dataflow.dflow DEBUG: Task 269 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 269 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 270 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 270 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 270 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 270 has memoization hash 41e6f77a022972d54320c3669b8847ae parsl.dataflow.memoization INFO: Task 270 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46837e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 270 try 0 launched on executor multi with executor id 91 parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/bd2aff70-d916-45a0-b649-f9c06405b49c_isr_855042_16.stdout parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/bd2aff70-d916-45a0-b649-f9c06405b49c_isr_855042_16.stderr parsl.dataflow.dflow DEBUG: Task 271 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 271 submitted for App characterizeImage, waiting on task 270 parsl.dataflow.dflow DEBUG: Task 271 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 271 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 272 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 272 submitted for App calibrate, waiting on task 271 parsl.dataflow.dflow DEBUG: Task 272 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 272 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 273 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 273 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 273 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 273 has memoization hash 1b245702c078f2b414b8f200a1f70a13 parsl.dataflow.memoization INFO: Task 273 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 273 try 0 launched on executor multi with executor id 92 parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/7804960a-684d-4bbf-997a-cb49efae53ba_isr_855136_26.stdout parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/7804960a-684d-4bbf-997a-cb49efae53ba_isr_855136_26.stderr parsl.dataflow.dflow DEBUG: Task 274 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 274 submitted for App characterizeImage, waiting on task 273 parsl.dataflow.dflow DEBUG: Task 274 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 274 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 275 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 275 submitted for App calibrate, waiting on task 274 parsl.dataflow.dflow DEBUG: Task 275 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 275 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 276 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 276 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 276 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 276 has memoization hash 8b60bbbb983792035843eb6c9d9667c5 parsl.dataflow.memoization INFO: Task 276 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 276 try 0 launched on executor multi with executor id 93 parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/4a27895d-7adf-49dc-9881-0311f13fb2e5_isr_855137_36.stdout parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/4a27895d-7adf-49dc-9881-0311f13fb2e5_isr_855137_36.stderr parsl.dataflow.dflow DEBUG: Task 277 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 277 submitted for App characterizeImage, waiting on task 276 parsl.dataflow.dflow DEBUG: Task 277 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 277 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 278 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 278 submitted for App calibrate, waiting on task 277 parsl.dataflow.dflow DEBUG: Task 278 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 278 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 279 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 279 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 279 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 279 has memoization hash e3b71e5c17274f11355a547410444ff3 parsl.dataflow.memoization INFO: Task 279 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 279 try 0 launched on executor multi with executor id 94 parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/b354673f-214e-4c94-a1c5-9fb7ee292d0d_isr_855044_1.stdout parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/b354673f-214e-4c94-a1c5-9fb7ee292d0d_isr_855044_1.stderr parsl.dataflow.dflow DEBUG: Task 280 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 280 submitted for App characterizeImage, waiting on task 279 parsl.dataflow.dflow DEBUG: Task 280 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 280 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 281 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 281 submitted for App calibrate, waiting on task 280 parsl.dataflow.dflow DEBUG: Task 281 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 281 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 282 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 282 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 282 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 282 has memoization hash 8833682497ceb7d0a9b154dba10eeb7f parsl.dataflow.memoization INFO: Task 282 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 282 try 0 launched on executor multi with executor id 95 parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/7a899baf-353e-4b0f-8e09-6872269e794b_isr_855043_36.stdout parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/7a899baf-353e-4b0f-8e09-6872269e794b_isr_855043_36.stderr parsl.dataflow.dflow DEBUG: Task 283 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 283 submitted for App characterizeImage, waiting on task 282 parsl.dataflow.dflow DEBUG: Task 283 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 283 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 284 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 284 submitted for App calibrate, waiting on task 283 parsl.dataflow.dflow DEBUG: Task 284 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 284 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 285 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 285 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 285 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 285 has memoization hash b5b173e91d4d2c407e89ca7891e44766 parsl.dataflow.memoization INFO: Task 285 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 285 try 0 launched on executor multi with executor id 96 parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/9d629c1b-93c1-47fb-a76e-d4f67543e9e4_isr_855137_29.stdout parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/9d629c1b-93c1-47fb-a76e-d4f67543e9e4_isr_855137_29.stderr parsl.dataflow.dflow DEBUG: Task 286 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 286 submitted for App characterizeImage, waiting on task 285 parsl.dataflow.dflow DEBUG: Task 286 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 286 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 287 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 287 submitted for App calibrate, waiting on task 286 parsl.dataflow.dflow DEBUG: Task 287 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 287 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 288 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 288 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 288 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 288 has memoization hash 03b31bebf768d48ce8a2794197395af4 parsl.dataflow.memoization INFO: Task 288 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 288 try 0 launched on executor multi with executor id 97 parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/c17cb89b-dff2-4716-9bcd-ca79783d2105_isr_855138_13.stdout parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/c17cb89b-dff2-4716-9bcd-ca79783d2105_isr_855138_13.stderr parsl.dataflow.dflow DEBUG: Task 289 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 289 submitted for App characterizeImage, waiting on task 288 parsl.dataflow.dflow DEBUG: Task 289 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 289 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 290 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 290 submitted for App calibrate, waiting on task 289 parsl.dataflow.dflow DEBUG: Task 290 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 290 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 291 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 291 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 291 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 291 has memoization hash c4ef731e764f3589441608c7624b92ef parsl.dataflow.memoization INFO: Task 291 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 291 try 0 launched on executor multi with executor id 98 parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/19130f1e-5ef1-496a-91dc-6dcb75ff9e92_isr_855044_12.stdout parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/19130f1e-5ef1-496a-91dc-6dcb75ff9e92_isr_855044_12.stderr parsl.dataflow.dflow DEBUG: Task 292 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 292 submitted for App characterizeImage, waiting on task 291 parsl.dataflow.dflow DEBUG: Task 292 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 292 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 293 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 293 submitted for App calibrate, waiting on task 292 parsl.dataflow.dflow DEBUG: Task 293 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 293 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 294 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 294 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 294 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 294 has memoization hash 95e4bf521916d6dfce561ca98d538f7a parsl.dataflow.memoization INFO: Task 294 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 294 try 0 launched on executor multi with executor id 99 parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/61f440f9-caad-47ad-aba7-6d0f224d1c6d_isr_855138_45.stdout parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/61f440f9-caad-47ad-aba7-6d0f224d1c6d_isr_855138_45.stderr parsl.dataflow.dflow DEBUG: Task 295 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 295 submitted for App characterizeImage, waiting on task 294 parsl.dataflow.dflow DEBUG: Task 295 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 295 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 296 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 296 submitted for App calibrate, waiting on task 295 parsl.dataflow.dflow DEBUG: Task 296 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 296 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 297 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 297 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 297 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 297 has memoization hash d06a5831e386b28ac99d30c6deb61fa2 parsl.dataflow.memoization INFO: Task 297 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 297 try 0 launched on executor multi with executor id 100 parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/eaa29b7d-dc64-41c5-810f-f6b2e4c11b8d_isr_855137_43.stdout parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/eaa29b7d-dc64-41c5-810f-f6b2e4c11b8d_isr_855137_43.stderr parsl.dataflow.dflow DEBUG: Task 298 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 298 submitted for App characterizeImage, waiting on task 297 parsl.dataflow.dflow DEBUG: Task 298 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 298 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 299 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 299 submitted for App calibrate, waiting on task 298 parsl.dataflow.dflow DEBUG: Task 299 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 299 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 300 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 300 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 300 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 300 has memoization hash 1067c9c6bd9ba88b4519cd6ace51bdcb parsl.dataflow.memoization INFO: Task 300 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 300 try 0 launched on executor multi with executor id 101 parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/c4786fdf-2466-4bee-b5ac-05b8665c125a_isr_855138_53.stdout parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/c4786fdf-2466-4bee-b5ac-05b8665c125a_isr_855138_53.stderr parsl.dataflow.dflow DEBUG: Task 301 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 301 submitted for App characterizeImage, waiting on task 300 parsl.dataflow.dflow DEBUG: Task 301 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 301 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 302 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 302 submitted for App calibrate, waiting on task 301 parsl.dataflow.dflow DEBUG: Task 302 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 302 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 303 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 303 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 303 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 303 has memoization hash 05a30b1d584a71b262571f01884274ac parsl.dataflow.memoization INFO: Task 303 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 303 try 0 launched on executor multi with executor id 102 parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/0d37e399-d0b5-4285-b29a-fee8bb9f55dc_isr_855044_39.stdout parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/0d37e399-d0b5-4285-b29a-fee8bb9f55dc_isr_855044_39.stderr parsl.dataflow.dflow DEBUG: Task 304 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 304 submitted for App characterizeImage, waiting on task 303 parsl.dataflow.dflow DEBUG: Task 304 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 304 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 305 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 305 submitted for App calibrate, waiting on task 304 parsl.dataflow.dflow DEBUG: Task 305 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 305 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 306 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 306 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 306 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 306 has memoization hash b25f0ec7e57dd6b1edaeb986f485e438 parsl.dataflow.memoization INFO: Task 306 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 306 try 0 launched on executor multi with executor id 103 parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/1b70c162-3ee5-472e-9490-de73cddc08ad_isr_855042_4.stdout parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/1b70c162-3ee5-472e-9490-de73cddc08ad_isr_855042_4.stderr parsl.dataflow.dflow DEBUG: Task 307 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 307 submitted for App characterizeImage, waiting on task 306 parsl.dataflow.dflow DEBUG: Task 307 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 307 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 308 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 308 submitted for App calibrate, waiting on task 307 parsl.dataflow.dflow DEBUG: Task 308 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 308 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 309 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 309 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 309 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 309 has memoization hash 4e1365ab112194bb27e48dec2bde3c7b parsl.dataflow.memoization INFO: Task 309 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 309 try 0 launched on executor multi with executor id 104 parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/ea61665c-dd14-4a61-98c6-a60446d78eeb_isr_855043_42.stdout parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/ea61665c-dd14-4a61-98c6-a60446d78eeb_isr_855043_42.stderr parsl.dataflow.dflow DEBUG: Task 310 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 310 submitted for App characterizeImage, waiting on task 309 parsl.dataflow.dflow DEBUG: Task 310 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 310 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 311 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 311 submitted for App calibrate, waiting on task 310 parsl.dataflow.dflow DEBUG: Task 311 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 311 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 312 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 312 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 312 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 312 has memoization hash 30fc1a1393ec54bc4b9e98d5c54fbe70 parsl.dataflow.memoization INFO: Task 312 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 312 try 0 launched on executor multi with executor id 105 parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/f5461997-dac6-426a-b509-e5007d08cea8_isr_855042_31.stdout parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/f5461997-dac6-426a-b509-e5007d08cea8_isr_855042_31.stderr parsl.dataflow.dflow DEBUG: Task 313 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 313 submitted for App characterizeImage, waiting on task 312 parsl.dataflow.dflow DEBUG: Task 313 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 313 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 314 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 314 submitted for App calibrate, waiting on task 313 parsl.dataflow.dflow DEBUG: Task 314 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 314 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 315 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 315 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 315 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 315 has memoization hash f4cb24fa53dfcb497868d112ce277104 parsl.dataflow.memoization INFO: Task 315 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 315 try 0 launched on executor multi with executor id 106 parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/06a77c3f-b51f-4ede-8400-ed388ade5349_isr_855043_26.stdout parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/06a77c3f-b51f-4ede-8400-ed388ade5349_isr_855043_26.stderr parsl.dataflow.dflow DEBUG: Task 316 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 316 submitted for App characterizeImage, waiting on task 315 parsl.dataflow.dflow DEBUG: Task 316 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 316 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 317 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 317 submitted for App calibrate, waiting on task 316 parsl.dataflow.dflow DEBUG: Task 317 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 317 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 318 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 318 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 318 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 318 has memoization hash 3caa219728dff3903411dde8b6606ab1 parsl.dataflow.memoization INFO: Task 318 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 318 try 0 launched on executor multi with executor id 107 parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/657462b1-eecf-4d26-aee0-b6d3a5acd0f0_isr_855044_56.stdout parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/657462b1-eecf-4d26-aee0-b6d3a5acd0f0_isr_855044_56.stderr parsl.dataflow.dflow DEBUG: Task 319 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 319 submitted for App characterizeImage, waiting on task 318 parsl.dataflow.dflow DEBUG: Task 319 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 319 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 320 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 320 submitted for App calibrate, waiting on task 319 parsl.dataflow.dflow DEBUG: Task 320 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 320 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 321 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 321 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 321 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 321 has memoization hash 3b8df6ab361dc027a924b6a59f44e81f parsl.dataflow.memoization INFO: Task 321 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 321 try 0 launched on executor multi with executor id 108 parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/63f481db-e270-47c3-a287-98fc9d25440b_isr_855042_28.stdout parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/63f481db-e270-47c3-a287-98fc9d25440b_isr_855042_28.stderr parsl.dataflow.dflow DEBUG: Task 322 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 322 submitted for App characterizeImage, waiting on task 321 parsl.dataflow.dflow DEBUG: Task 322 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 322 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 323 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 323 submitted for App calibrate, waiting on task 322 parsl.dataflow.dflow DEBUG: Task 323 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 323 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 324 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 324 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 324 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 324 has memoization hash 44912b1a2e49e457caf73ab947906abc parsl.dataflow.memoization INFO: Task 324 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 324 try 0 launched on executor multi with executor id 109 parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/5ff64e32-dd17-400f-b8f3-8eec0a9a61b0_isr_855042_52.stdout parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/5ff64e32-dd17-400f-b8f3-8eec0a9a61b0_isr_855042_52.stderr parsl.dataflow.dflow DEBUG: Task 325 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 325 submitted for App characterizeImage, waiting on task 324 parsl.dataflow.dflow DEBUG: Task 325 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 325 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 326 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 326 submitted for App calibrate, waiting on task 325 parsl.dataflow.dflow DEBUG: Task 326 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 326 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 327 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 327 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 327 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 327 has memoization hash 9658da980f64c9d3eca07a333344a606 parsl.dataflow.memoization INFO: Task 327 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 327 try 0 launched on executor multi with executor id 110 parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/68543290-cc20-45dc-a640-55869cb9834b_isr_855138_58.stdout parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/68543290-cc20-45dc-a640-55869cb9834b_isr_855138_58.stderr parsl.dataflow.dflow DEBUG: Task 328 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 328 submitted for App characterizeImage, waiting on task 327 parsl.dataflow.dflow DEBUG: Task 328 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 328 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 329 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 329 submitted for App calibrate, waiting on task 328 parsl.dataflow.dflow DEBUG: Task 329 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 329 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 330 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 330 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 330 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 330 has memoization hash 40838ad3fffafff7385b0f3f672eb2de parsl.dataflow.memoization INFO: Task 330 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 330 try 0 launched on executor multi with executor id 111 parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/318800a0-6f2b-4305-a079-1b210594fa13_isr_855043_62.stdout parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/318800a0-6f2b-4305-a079-1b210594fa13_isr_855043_62.stderr parsl.dataflow.dflow DEBUG: Task 331 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 331 submitted for App characterizeImage, waiting on task 330 parsl.dataflow.dflow DEBUG: Task 331 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 331 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 332 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 332 submitted for App calibrate, waiting on task 331 parsl.dataflow.dflow DEBUG: Task 332 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 332 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 333 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 333 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 333 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 333 has memoization hash f6f3f783e69f6f02104819207ca38e88 parsl.dataflow.memoization INFO: Task 333 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46878040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 333 try 0 launched on executor multi with executor id 112 parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/03a7c97d-77f1-473e-bd5a-4b10ad17405a_isr_855138_24.stdout parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/03a7c97d-77f1-473e-bd5a-4b10ad17405a_isr_855138_24.stderr parsl.dataflow.dflow DEBUG: Task 334 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 334 submitted for App characterizeImage, waiting on task 333 parsl.dataflow.dflow DEBUG: Task 334 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 334 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 335 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 335 submitted for App calibrate, waiting on task 334 parsl.dataflow.dflow DEBUG: Task 335 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 335 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 336 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 336 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 336 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 336 has memoization hash 154d55cbf959fecb7d299785ec11041a parsl.dataflow.memoization INFO: Task 336 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46878360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 336 try 0 launched on executor multi with executor id 113 parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/5def27a8-b097-40bc-9bbf-5a078a34e457_isr_855043_5.stdout parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/5def27a8-b097-40bc-9bbf-5a078a34e457_isr_855043_5.stderr parsl.dataflow.dflow DEBUG: Task 337 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 337 submitted for App characterizeImage, waiting on task 336 parsl.dataflow.dflow DEBUG: Task 337 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 337 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 338 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 338 submitted for App calibrate, waiting on task 337 parsl.dataflow.dflow DEBUG: Task 338 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 338 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 339 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 339 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 339 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 339 has memoization hash d9d430831c13ff1a7d8f7549c1e09d0c parsl.dataflow.memoization INFO: Task 339 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46878680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 339 try 0 launched on executor multi with executor id 114 parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/51f3d2b2-e19a-4348-9333-5b5bf3449001_isr_855137_47.stdout parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/51f3d2b2-e19a-4348-9333-5b5bf3449001_isr_855137_47.stderr parsl.dataflow.dflow DEBUG: Task 340 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 340 submitted for App characterizeImage, waiting on task 339 parsl.dataflow.dflow DEBUG: Task 340 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 340 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 341 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 341 submitted for App calibrate, waiting on task 340 parsl.dataflow.dflow DEBUG: Task 341 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 341 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 342 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 342 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 342 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 342 has memoization hash 9d50d73da6f692c2c31aa4164d1850d6 parsl.dataflow.memoization INFO: Task 342 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa468789a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 342 try 0 launched on executor multi with executor id 115 parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/c24a396a-46ba-4584-b462-ff95a89a1e13_isr_855136_59.stdout parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/c24a396a-46ba-4584-b462-ff95a89a1e13_isr_855136_59.stderr parsl.dataflow.dflow DEBUG: Task 343 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 343 submitted for App characterizeImage, waiting on task 342 parsl.dataflow.dflow DEBUG: Task 343 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 343 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 344 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 344 submitted for App calibrate, waiting on task 343 parsl.dataflow.dflow DEBUG: Task 344 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 344 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 345 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 345 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 345 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 345 has memoization hash 8819ecdd3547b4ebb31079f6efdefb26 parsl.dataflow.memoization INFO: Task 345 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46878cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 345 try 0 launched on executor multi with executor id 116 parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/ea5cde7a-91f6-4106-9a40-8dac767c91ff_isr_855043_25.stdout parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/ea5cde7a-91f6-4106-9a40-8dac767c91ff_isr_855043_25.stderr parsl.dataflow.dflow DEBUG: Task 346 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 346 submitted for App characterizeImage, waiting on task 345 parsl.dataflow.dflow DEBUG: Task 346 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 346 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 347 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 347 submitted for App calibrate, waiting on task 346 parsl.dataflow.dflow DEBUG: Task 347 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 347 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 348 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 348 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 348 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 348 has memoization hash 25057dcf7b75e65a457299685901ee79 parsl.dataflow.memoization INFO: Task 348 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46878fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 348 try 0 launched on executor multi with executor id 117 parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/68c3986f-914a-46de-b21e-c11918654365_isr_855136_30.stdout parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/68c3986f-914a-46de-b21e-c11918654365_isr_855136_30.stderr parsl.dataflow.dflow DEBUG: Task 349 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 349 submitted for App characterizeImage, waiting on task 348 parsl.dataflow.dflow DEBUG: Task 349 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 349 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 350 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 350 submitted for App calibrate, waiting on task 349 parsl.dataflow.dflow DEBUG: Task 350 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 350 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 351 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 351 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 351 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 351 has memoization hash 5a725b47e7c2b27eca9cb965423ee502 parsl.dataflow.memoization INFO: Task 351 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46879300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 351 try 0 launched on executor multi with executor id 118 parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/d4dcfe6f-ad44-4575-95ab-41ffd1a42e95_isr_855042_38.stdout parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/d4dcfe6f-ad44-4575-95ab-41ffd1a42e95_isr_855042_38.stderr parsl.dataflow.dflow DEBUG: Task 352 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 352 submitted for App characterizeImage, waiting on task 351 parsl.dataflow.dflow DEBUG: Task 352 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 352 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 353 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 353 submitted for App calibrate, waiting on task 352 parsl.dataflow.dflow DEBUG: Task 353 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 353 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 354 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 354 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 354 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 354 has memoization hash 3f41463435d3a1a5d9dc63352ef35c28 parsl.dataflow.memoization INFO: Task 354 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46879620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 354 try 0 launched on executor multi with executor id 119 parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/8a84493d-d932-4f4e-8ab9-8bc8b57a47e7_isr_855138_10.stdout parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/8a84493d-d932-4f4e-8ab9-8bc8b57a47e7_isr_855138_10.stderr parsl.dataflow.dflow DEBUG: Task 355 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 355 submitted for App characterizeImage, waiting on task 354 parsl.dataflow.dflow DEBUG: Task 355 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 355 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 356 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 356 submitted for App calibrate, waiting on task 355 parsl.dataflow.dflow DEBUG: Task 356 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 356 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 357 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 357 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 357 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 357 has memoization hash 4b2431a59f7c5e95786a3c8fbbb94bc0 parsl.dataflow.memoization INFO: Task 357 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46879940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 357 try 0 launched on executor multi with executor id 120 parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/237014a8-6b44-4d95-910a-dbf483d575d1_isr_855042_34.stdout parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/237014a8-6b44-4d95-910a-dbf483d575d1_isr_855042_34.stderr parsl.dataflow.dflow DEBUG: Task 358 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 358 submitted for App characterizeImage, waiting on task 357 parsl.dataflow.dflow DEBUG: Task 358 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 358 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 359 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 359 submitted for App calibrate, waiting on task 358 parsl.dataflow.dflow DEBUG: Task 359 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 359 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 360 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 360 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 360 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 360 has memoization hash 9ecda11f1f0d66ba0abd53315c8ecae1 parsl.dataflow.memoization INFO: Task 360 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46879c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 360 try 0 launched on executor multi with executor id 121 parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/bb4a6cb9-2152-4fcf-b12a-becdad1d33df_isr_855137_25.stdout parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/bb4a6cb9-2152-4fcf-b12a-becdad1d33df_isr_855137_25.stderr parsl.dataflow.dflow DEBUG: Task 361 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 361 submitted for App characterizeImage, waiting on task 360 parsl.dataflow.dflow DEBUG: Task 361 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 361 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 362 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 362 submitted for App calibrate, waiting on task 361 parsl.dataflow.dflow DEBUG: Task 362 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 362 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 363 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 363 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 363 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 363 has memoization hash 04a5ac1da77fedb3cd222e4f71c1f323 parsl.dataflow.memoization INFO: Task 363 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46879f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 363 try 0 launched on executor multi with executor id 122 parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/0a1b03ca-c85f-4862-adc1-9bfb68bc8f86_isr_855043_30.stdout parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/0a1b03ca-c85f-4862-adc1-9bfb68bc8f86_isr_855043_30.stderr parsl.dataflow.dflow DEBUG: Task 364 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 364 submitted for App characterizeImage, waiting on task 363 parsl.dataflow.dflow DEBUG: Task 364 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 364 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 365 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 365 submitted for App calibrate, waiting on task 364 parsl.dataflow.dflow DEBUG: Task 365 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 365 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 366 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 366 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 366 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 366 has memoization hash 6f972905be8aa134dd42e8fc6cc46124 parsl.dataflow.memoization INFO: Task 366 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4687a2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 366 try 0 launched on executor multi with executor id 123 parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/97b5307d-64bd-4d9a-afc0-248a428a35a5_isr_855044_17.stdout parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/97b5307d-64bd-4d9a-afc0-248a428a35a5_isr_855044_17.stderr parsl.dataflow.dflow DEBUG: Task 367 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 367 submitted for App characterizeImage, waiting on task 366 parsl.dataflow.dflow DEBUG: Task 367 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 367 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 368 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 368 submitted for App calibrate, waiting on task 367 parsl.dataflow.dflow DEBUG: Task 368 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 368 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 369 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 369 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 369 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 369 has memoization hash df80d5d596f8322b2e26c80af3749de9 parsl.dataflow.memoization INFO: Task 369 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4687a5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 369 try 0 launched on executor multi with executor id 124 parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/6d74f0ca-c255-4305-ad23-73539a9dea64_isr_855137_3.stdout parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/6d74f0ca-c255-4305-ad23-73539a9dea64_isr_855137_3.stderr parsl.dataflow.dflow DEBUG: Task 370 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 370 submitted for App characterizeImage, waiting on task 369 parsl.dataflow.dflow DEBUG: Task 370 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 370 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 371 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 371 submitted for App calibrate, waiting on task 370 parsl.dataflow.dflow DEBUG: Task 371 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 371 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 372 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 372 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 372 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 372 has memoization hash 6295370b6b200d25c462c0203cbb28c4 parsl.dataflow.memoization INFO: Task 372 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4687a8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-08 07:15:11 proc_lsst.multi:146 [INFO] found job 75775 in provider local proc_lsst.multi INFO: found job 75775 in provider local parsl.dataflow.dflow INFO: Parsl task 372 try 0 launched on executor multi with executor id 125 parsl.dataflow.dflow INFO: Standard output for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/ae16d5eb-0d3b-4e98-b48c-156ad007c5c6_isr_855043_47.stdout parsl.dataflow.dflow INFO: Standard error for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/ae16d5eb-0d3b-4e98-b48c-156ad007c5c6_isr_855043_47.stderr parsl.dataflow.dflow DEBUG: Task 373 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 373 submitted for App characterizeImage, waiting on task 372 parsl.dataflow.dflow DEBUG: Task 373 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 373 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 374 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 374 submitted for App calibrate, waiting on task 373 parsl.dataflow.dflow DEBUG: Task 374 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 374 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 375 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 375 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 375 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 375 has memoization hash 7a7c56066465395a26cbc1d050d3dc45 parsl.dataflow.memoization INFO: Task 375 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a5da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 375 try 0 launched on executor multi with executor id 126 parsl.dataflow.dflow INFO: Standard output for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/5f4fa116-e97e-4908-93ab-82e991f41268_isr_855042_29.stdout parsl.dataflow.dflow INFO: Standard error for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/5f4fa116-e97e-4908-93ab-82e991f41268_isr_855042_29.stderr parsl.dataflow.dflow DEBUG: Task 376 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 376 submitted for App characterizeImage, waiting on task 375 parsl.dataflow.dflow DEBUG: Task 376 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 376 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 377 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 377 submitted for App calibrate, waiting on task 376 parsl.dataflow.dflow DEBUG: Task 377 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 377 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 378 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 378 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 378 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 378 has memoization hash c49d6d3238ae9fc44f294e7b9e1be66a parsl.dataflow.memoization INFO: Task 378 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a5a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 378 try 0 launched on executor multi with executor id 127 parsl.dataflow.dflow INFO: Standard output for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/8f6c0d83-99ee-4e8f-ab40-d3259a812e2a_isr_855138_14.stdout parsl.dataflow.dflow INFO: Standard error for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/8f6c0d83-99ee-4e8f-ab40-d3259a812e2a_isr_855138_14.stderr parsl.dataflow.dflow DEBUG: Task 379 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 379 submitted for App characterizeImage, waiting on task 378 parsl.dataflow.dflow DEBUG: Task 379 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 379 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 380 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 380 submitted for App calibrate, waiting on task 379 parsl.dataflow.dflow DEBUG: Task 380 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 380 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 381 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 381 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 381 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 381 has memoization hash e979cf75ee5090df46477ec925413d1e parsl.dataflow.memoization INFO: Task 381 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4687afc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 381 try 0 launched on executor multi with executor id 128 parsl.dataflow.dflow INFO: Standard output for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/46ccc8e2-07bd-4a50-a91c-d68b6114983b_isr_855042_59.stdout parsl.dataflow.dflow INFO: Standard error for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/46ccc8e2-07bd-4a50-a91c-d68b6114983b_isr_855042_59.stderr parsl.dataflow.dflow DEBUG: Task 382 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 382 submitted for App characterizeImage, waiting on task 381 parsl.dataflow.dflow DEBUG: Task 382 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 382 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 383 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 383 submitted for App calibrate, waiting on task 382 parsl.dataflow.dflow DEBUG: Task 383 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 383 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 384 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 384 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 384 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 384 has memoization hash 8ddc72574d3b15e885c3c80c452d5635 parsl.dataflow.memoization INFO: Task 384 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4687b380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 384 try 0 launched on executor multi with executor id 129 parsl.dataflow.dflow INFO: Standard output for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/e0dd50af-f11d-4b1d-9956-5e98856219b9_isr_855136_1.stdout parsl.dataflow.dflow INFO: Standard error for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/e0dd50af-f11d-4b1d-9956-5e98856219b9_isr_855136_1.stderr parsl.dataflow.dflow DEBUG: Task 385 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 385 submitted for App characterizeImage, waiting on task 384 parsl.dataflow.dflow DEBUG: Task 385 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 385 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 386 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 386 submitted for App calibrate, waiting on task 385 parsl.dataflow.dflow DEBUG: Task 386 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 386 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 387 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 387 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 387 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 387 has memoization hash 3d91bc9a2938124bc6cd12ee26b3db86 parsl.dataflow.memoization INFO: Task 387 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4687b740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 387 try 0 launched on executor multi with executor id 130 parsl.dataflow.dflow INFO: Standard output for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/7b34fa48-0c3f-40b2-9af3-a582b7699d2e_isr_855042_51.stdout parsl.dataflow.dflow INFO: Standard error for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/7b34fa48-0c3f-40b2-9af3-a582b7699d2e_isr_855042_51.stderr parsl.dataflow.dflow DEBUG: Task 388 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 388 submitted for App characterizeImage, waiting on task 387 parsl.dataflow.dflow DEBUG: Task 388 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 388 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 389 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 389 submitted for App calibrate, waiting on task 388 parsl.dataflow.dflow DEBUG: Task 389 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 389 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 390 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 390 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 390 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 390 has memoization hash 88155f41c1482dbe8921c3beeb9458a0 parsl.dataflow.memoization INFO: Task 390 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4687ba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 390 try 0 launched on executor multi with executor id 131 parsl.dataflow.dflow INFO: Standard output for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/5250b605-9001-45d2-94e9-5126ee258c3a_isr_855137_2.stdout parsl.dataflow.dflow INFO: Standard error for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/5250b605-9001-45d2-94e9-5126ee258c3a_isr_855137_2.stderr parsl.dataflow.dflow DEBUG: Task 391 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 391 submitted for App characterizeImage, waiting on task 390 parsl.dataflow.dflow DEBUG: Task 391 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 391 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 392 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 392 submitted for App calibrate, waiting on task 391 parsl.dataflow.dflow DEBUG: Task 392 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 392 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 393 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 393 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 393 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 393 has memoization hash bb98bde4fa401f995efe052cefe9b677 parsl.dataflow.memoization INFO: Task 393 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4687bd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 393 try 0 launched on executor multi with executor id 132 parsl.dataflow.dflow INFO: Standard output for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/25e33014-2875-421a-9aac-71492f9e8a14_isr_855044_5.stdout parsl.dataflow.dflow INFO: Standard error for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/25e33014-2875-421a-9aac-71492f9e8a14_isr_855044_5.stderr parsl.dataflow.dflow DEBUG: Task 394 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 394 submitted for App characterizeImage, waiting on task 393 parsl.dataflow.dflow DEBUG: Task 394 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 394 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 395 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 395 submitted for App calibrate, waiting on task 394 parsl.dataflow.dflow DEBUG: Task 395 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 395 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 396 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 396 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 396 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 396 has memoization hash 98b517f34d427eac1391ccaa103e7d77 parsl.dataflow.memoization INFO: Task 396 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a58a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 132 parsl.dataflow.strategy DEBUG: Executor multi has 132 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 32 more blocks parsl.executors.status_handling INFO: Scaling out by 32 blocks parsl.executors.status_handling INFO: Allocated block ID 1 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1 2024-03-08 07:15:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 2024-03-08 07:15:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-08 07:15:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 396 try 0 launched on executor multi with executor id 133 parsl.dataflow.dflow INFO: Standard output for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/383e3cd7-b91a-4c25-9623-298eed50e1bb_isr_855138_48.stdout parsl.dataflow.dflow INFO: Standard error for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/383e3cd7-b91a-4c25-9623-298eed50e1bb_isr_855138_48.stderr parsl.dataflow.dflow DEBUG: Task 397 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 397 submitted for App characterizeImage, waiting on task 396 parsl.dataflow.dflow DEBUG: Task 397 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 397 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 398 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 398 submitted for App calibrate, waiting on task 397 parsl.dataflow.dflow DEBUG: Task 398 set to pending state with AppFuture: parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 398 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 399 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 399 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 399 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 399 has memoization hash 1a78cf4e2dd041e6cbec1b0f63065607 parsl.dataflow.memoization INFO: Task 399 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a4040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 399 try 0 launched on executor multi with executor id 134 parsl.dataflow.dflow INFO: Standard output for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/ff291be9-e449-4de5-93ad-ac988fbb938a_isr_855137_27.stdout parsl.dataflow.dflow INFO: Standard error for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/ff291be9-e449-4de5-93ad-ac988fbb938a_isr_855137_27.stderr parsl.dataflow.dflow DEBUG: Task 400 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 400 submitted for App characterizeImage, waiting on task 399 parsl.dataflow.dflow DEBUG: Task 400 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 400 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 401 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 401 submitted for App calibrate, waiting on task 400 parsl.dataflow.dflow DEBUG: Task 401 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 401 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 402 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 402 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 402 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 402 has memoization hash 04ea167d93e64eceafa0ffea65cbcc5c parsl.dataflow.memoization INFO: Task 402 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a7920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 402 try 0 launched on executor multi with executor id 135 parsl.dataflow.dflow INFO: Standard output for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/c6897742-65df-49f4-b23a-ea76e7ec1327_isr_855136_43.stdout parsl.dataflow.dflow INFO: Standard error for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/c6897742-65df-49f4-b23a-ea76e7ec1327_isr_855136_43.stderr parsl.dataflow.dflow DEBUG: Task 403 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 403 submitted for App characterizeImage, waiting on task 402 parsl.dataflow.dflow DEBUG: Task 403 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 403 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 404 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 404 submitted for App calibrate, waiting on task 403 parsl.dataflow.dflow DEBUG: Task 404 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 404 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 405 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 405 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 405 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 405 has memoization hash bcaa42173e72e319b330baa3cfb8fb9f parsl.dataflow.memoization INFO: Task 405 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a7600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 405 try 0 launched on executor multi with executor id 136 parsl.dataflow.dflow INFO: Standard output for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/2a0f9084-0715-4aba-8314-243abbb1f521_isr_855137_45.stdout parsl.dataflow.dflow INFO: Standard error for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/2a0f9084-0715-4aba-8314-243abbb1f521_isr_855137_45.stderr parsl.dataflow.dflow DEBUG: Task 406 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 406 submitted for App characterizeImage, waiting on task 405 parsl.dataflow.dflow DEBUG: Task 406 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 406 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 407 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 407 submitted for App calibrate, waiting on task 406 parsl.dataflow.dflow DEBUG: Task 407 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 407 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 408 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 408 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 408 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 408 has memoization hash a15e8b5673c82965c2899df2b2d84126 parsl.dataflow.memoization INFO: Task 408 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a72e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 408 try 0 launched on executor multi with executor id 137 parsl.dataflow.dflow INFO: Standard output for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/ce4cd375-67f5-4bfd-ab0f-c9c57797dae2_isr_855043_48.stdout parsl.dataflow.dflow INFO: Standard error for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/ce4cd375-67f5-4bfd-ab0f-c9c57797dae2_isr_855043_48.stderr parsl.dataflow.dflow DEBUG: Task 409 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 409 submitted for App characterizeImage, waiting on task 408 parsl.dataflow.dflow DEBUG: Task 409 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 409 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 410 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 410 submitted for App calibrate, waiting on task 409 parsl.dataflow.dflow DEBUG: Task 410 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 410 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 411 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 411 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 411 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 411 has memoization hash 8a2eb5e56d261937a166ccf22f341397 parsl.dataflow.memoization INFO: Task 411 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a6fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 411 try 0 launched on executor multi with executor id 138 parsl.dataflow.dflow INFO: Standard output for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/f164302b-5e1b-4175-97fd-b1e19adf0f1b_isr_855136_38.stdout parsl.dataflow.dflow INFO: Standard error for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/f164302b-5e1b-4175-97fd-b1e19adf0f1b_isr_855136_38.stderr parsl.dataflow.dflow DEBUG: Task 412 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 412 submitted for App characterizeImage, waiting on task 411 parsl.dataflow.dflow DEBUG: Task 412 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 412 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 413 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 413 submitted for App calibrate, waiting on task 412 parsl.dataflow.dflow DEBUG: Task 413 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 413 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 414 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 414 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 414 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 414 has memoization hash 99bf0eccd0b9896a5e192ffd6463babb parsl.dataflow.memoization INFO: Task 414 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a6ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 414 try 0 launched on executor multi with executor id 139 parsl.dataflow.dflow INFO: Standard output for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/2c565da0-50df-4160-b487-d12b1ed06ff4_isr_855042_12.stdout parsl.dataflow.dflow INFO: Standard error for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/2c565da0-50df-4160-b487-d12b1ed06ff4_isr_855042_12.stderr parsl.dataflow.dflow DEBUG: Task 415 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 415 submitted for App characterizeImage, waiting on task 414 parsl.dataflow.dflow DEBUG: Task 415 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 415 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 416 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 416 submitted for App calibrate, waiting on task 415 parsl.dataflow.dflow DEBUG: Task 416 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 416 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 417 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 417 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 417 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 417 has memoization hash 7a963f419d03d5d648445378946a4c6d parsl.dataflow.memoization INFO: Task 417 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a42c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 417 try 0 launched on executor multi with executor id 140 parsl.dataflow.dflow INFO: Standard output for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/de8a68b1-ce98-4db0-8e04-c48db6703d53_isr_855137_59.stdout parsl.dataflow.dflow INFO: Standard error for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/de8a68b1-ce98-4db0-8e04-c48db6703d53_isr_855137_59.stderr parsl.dataflow.dflow DEBUG: Task 418 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 418 submitted for App characterizeImage, waiting on task 417 parsl.dataflow.dflow DEBUG: Task 418 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 418 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 419 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 419 submitted for App calibrate, waiting on task 418 parsl.dataflow.dflow DEBUG: Task 419 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 419 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 420 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 420 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 420 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 420 has memoization hash d960c5fa05d444e71aeacc2848f0b272 parsl.dataflow.memoization INFO: Task 420 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a49a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 420 try 0 launched on executor multi with executor id 141 parsl.dataflow.dflow INFO: Standard output for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/7bbeab17-91cd-4a10-a2e9-0a88c4fb4a71_isr_855136_28.stdout parsl.dataflow.dflow INFO: Standard error for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/7bbeab17-91cd-4a10-a2e9-0a88c4fb4a71_isr_855136_28.stderr parsl.dataflow.dflow DEBUG: Task 421 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 421 submitted for App characterizeImage, waiting on task 420 parsl.dataflow.dflow DEBUG: Task 421 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 421 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 422 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 422 submitted for App calibrate, waiting on task 421 parsl.dataflow.dflow DEBUG: Task 422 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 422 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 423 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 423 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 423 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 423 has memoization hash 6ef5506ed50f749b0bae771bcedce166 parsl.dataflow.memoization INFO: Task 423 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a5800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 423 try 0 launched on executor multi with executor id 142 parsl.dataflow.dflow INFO: Standard output for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/e4bbdce0-0896-41b1-9aaf-85ebd3a09d52_isr_855044_51.stdout parsl.dataflow.dflow INFO: Standard error for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/e4bbdce0-0896-41b1-9aaf-85ebd3a09d52_isr_855044_51.stderr parsl.dataflow.dflow DEBUG: Task 424 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 424 submitted for App characterizeImage, waiting on task 423 parsl.dataflow.dflow DEBUG: Task 424 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 424 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 425 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 425 submitted for App calibrate, waiting on task 424 parsl.dataflow.dflow DEBUG: Task 425 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 425 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 426 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 426 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 426 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 426 has memoization hash e206d2b3846cb3cb5439739613676aa8 parsl.dataflow.memoization INFO: Task 426 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a54e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 426 try 0 launched on executor multi with executor id 143 parsl.dataflow.dflow INFO: Standard output for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/0127fa18-ffa9-423a-96ed-d782e7ec314f_isr_855136_25.stdout parsl.dataflow.dflow INFO: Standard error for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/0127fa18-ffa9-423a-96ed-d782e7ec314f_isr_855136_25.stderr parsl.dataflow.dflow DEBUG: Task 427 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 427 submitted for App characterizeImage, waiting on task 426 parsl.dataflow.dflow DEBUG: Task 427 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 427 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 428 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 428 submitted for App calibrate, waiting on task 427 parsl.dataflow.dflow DEBUG: Task 428 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 428 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 429 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 429 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 429 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 429 has memoization hash 6574ea7554dccaa14de3ab8775118c2a parsl.dataflow.memoization INFO: Task 429 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a5300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 429 try 0 launched on executor multi with executor id 144 parsl.dataflow.dflow INFO: Standard output for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/70d759b5-0af7-441a-9f2b-dbc045f69a39_isr_855043_4.stdout parsl.dataflow.dflow INFO: Standard error for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/70d759b5-0af7-441a-9f2b-dbc045f69a39_isr_855043_4.stderr parsl.dataflow.dflow DEBUG: Task 430 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 430 submitted for App characterizeImage, waiting on task 429 parsl.dataflow.dflow DEBUG: Task 430 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 430 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 431 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 431 submitted for App calibrate, waiting on task 430 parsl.dataflow.dflow DEBUG: Task 431 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 431 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 432 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 432 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 432 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 432 has memoization hash e4366fcfa86fd9aa28bfd84a27f1b693 parsl.dataflow.memoization INFO: Task 432 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a4fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 432 try 0 launched on executor multi with executor id 145 parsl.dataflow.dflow INFO: Standard output for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/9f3da9cb-1833-4e0c-ad16-c2a20a95d29e_isr_855042_27.stdout parsl.dataflow.dflow INFO: Standard error for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/9f3da9cb-1833-4e0c-ad16-c2a20a95d29e_isr_855042_27.stderr parsl.dataflow.dflow DEBUG: Task 433 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 433 submitted for App characterizeImage, waiting on task 432 parsl.dataflow.dflow DEBUG: Task 433 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 433 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 434 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 434 submitted for App calibrate, waiting on task 433 parsl.dataflow.dflow DEBUG: Task 434 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 434 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 435 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 435 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 435 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 435 has memoization hash c58a30940c8723320836483b892c64c4 parsl.dataflow.memoization INFO: Task 435 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a4cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 435 try 0 launched on executor multi with executor id 146 parsl.dataflow.dflow INFO: Standard output for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/55a1bd20-2fe8-4b30-ac1c-ceb79006b994_isr_855137_14.stdout parsl.dataflow.dflow INFO: Standard error for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/55a1bd20-2fe8-4b30-ac1c-ceb79006b994_isr_855137_14.stderr parsl.dataflow.dflow DEBUG: Task 436 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 436 submitted for App characterizeImage, waiting on task 435 parsl.dataflow.dflow DEBUG: Task 436 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 436 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 437 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 437 submitted for App calibrate, waiting on task 436 parsl.dataflow.dflow DEBUG: Task 437 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 437 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 438 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 438 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 438 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 438 has memoization hash aa5076301b92ac5f87297b713edffc8f parsl.dataflow.memoization INFO: Task 438 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a62a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 438 try 0 launched on executor multi with executor id 147 parsl.dataflow.dflow INFO: Standard output for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/ae8ce82a-fa7c-41ed-ab45-fb37599feb99_isr_855044_23.stdout parsl.dataflow.dflow INFO: Standard error for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/ae8ce82a-fa7c-41ed-ab45-fb37599feb99_isr_855044_23.stderr parsl.dataflow.dflow DEBUG: Task 439 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 439 submitted for App characterizeImage, waiting on task 438 parsl.dataflow.dflow DEBUG: Task 439 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 439 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 440 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 440 submitted for App calibrate, waiting on task 439 parsl.dataflow.dflow DEBUG: Task 440 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 440 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 441 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 441 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 441 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 441 has memoization hash 69442ead886187dc030904a3b1a736df parsl.dataflow.memoization INFO: Task 441 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a65c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 441 try 0 launched on executor multi with executor id 148 parsl.dataflow.dflow INFO: Standard output for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/adb78243-5192-4ca2-a800-0e61041dadf5_isr_855138_1.stdout parsl.dataflow.dflow INFO: Standard error for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/adb78243-5192-4ca2-a800-0e61041dadf5_isr_855138_1.stderr parsl.dataflow.dflow DEBUG: Task 442 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 442 submitted for App characterizeImage, waiting on task 441 parsl.dataflow.dflow DEBUG: Task 442 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 442 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 443 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 443 submitted for App calibrate, waiting on task 442 parsl.dataflow.dflow DEBUG: Task 443 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 443 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 444 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 444 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 444 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 444 has memoization hash 9d1ebb4e6c48500dec131da1bce8b113 parsl.dataflow.memoization INFO: Task 444 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a6d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 444 try 0 launched on executor multi with executor id 149 parsl.dataflow.dflow INFO: Standard output for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/b7e47673-7ccc-4113-8bfa-4fd66e3ab26c_isr_855043_1.stdout parsl.dataflow.dflow INFO: Standard error for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/b7e47673-7ccc-4113-8bfa-4fd66e3ab26c_isr_855043_1.stderr parsl.dataflow.dflow DEBUG: Task 445 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 445 submitted for App characterizeImage, waiting on task 444 parsl.dataflow.dflow DEBUG: Task 445 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 445 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 446 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 446 submitted for App calibrate, waiting on task 445 parsl.dataflow.dflow DEBUG: Task 446 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 446 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 447 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 447 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 447 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 447 has memoization hash 753a02baa1f6067cad4894dedf6495d7 parsl.dataflow.memoization INFO: Task 447 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a6980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 447 try 0 launched on executor multi with executor id 150 parsl.dataflow.dflow INFO: Standard output for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/47867cb3-4d13-43de-b532-7cf719e45c2f_isr_855137_49.stdout parsl.dataflow.dflow INFO: Standard error for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/47867cb3-4d13-43de-b532-7cf719e45c2f_isr_855137_49.stderr parsl.dataflow.dflow DEBUG: Task 448 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 448 submitted for App characterizeImage, waiting on task 447 parsl.dataflow.dflow DEBUG: Task 448 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 448 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 449 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 449 submitted for App calibrate, waiting on task 448 2024-03-08 07:15:12 proc_lsst.multi:170 [INFO] [multi] job_id 16912283 proc_lsst.multi INFO: [multi] job_id 16912283 2024-03-08 07:15:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:12 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16912283 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16912283 parsl.dataflow.dflow DEBUG: Task 449 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 449 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 450 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 450 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 450 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 450 has memoization hash 24242c8da71013e68b3c5c04f093d674 parsl.dataflow.memoization INFO: Task 450 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a4180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.executors.status_handling DEBUG: Launched block 1 on executor multi with job ID 16912283 parsl.dataflow.dflow INFO: Parsl task 450 try 0 launched on executor multi with executor id 151 parsl.dataflow.dflow INFO: Standard output for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/faa5da83-1dff-4b2d-85ee-b39730ddf98c_isr_855138_9.stdout parsl.dataflow.dflow INFO: Standard error for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/faa5da83-1dff-4b2d-85ee-b39730ddf98c_isr_855138_9.stderr parsl.dataflow.dflow DEBUG: Task 451 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 451 submitted for App characterizeImage, waiting on task 450 parsl.dataflow.dflow DEBUG: Task 451 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 451 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 452 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 452 submitted for App calibrate, waiting on task 451 parsl.dataflow.dflow DEBUG: Task 452 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 452 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 453 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 453 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 453 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 453 has memoization hash 6074fd70655549751230b0ae1f1aee60 parsl.dataflow.memoization INFO: Task 453 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a7e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.executors.status_handling INFO: Allocated block ID 2 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2 2024-03-08 07:15:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 2024-03-08 07:15:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 453 try 0 launched on executor multi with executor id 152 parsl.dataflow.dflow INFO: Standard output for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/cb372d8f-c9da-4cfa-9113-c3923f0f5b43_isr_855137_20.stdout parsl.dataflow.dflow INFO: Standard error for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/cb372d8f-c9da-4cfa-9113-c3923f0f5b43_isr_855137_20.stderr parsl.dataflow.dflow DEBUG: Task 454 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 454 submitted for App characterizeImage, waiting on task 453 parsl.dataflow.dflow DEBUG: Task 454 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 454 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 455 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 455 submitted for App calibrate, waiting on task 454 parsl.dataflow.dflow DEBUG: Task 455 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 455 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 456 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 456 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 456 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 456 has memoization hash 138523e560631bc9a05db2e8a6bf16a3 parsl.dataflow.memoization INFO: Task 456 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456cc2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 456 try 0 launched on executor multi with executor id 153 parsl.dataflow.dflow INFO: Standard output for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/cfb5f3f1-7cda-4d80-b6d3-76bb8e378715_isr_855136_4.stdout parsl.dataflow.dflow INFO: Standard error for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/cfb5f3f1-7cda-4d80-b6d3-76bb8e378715_isr_855136_4.stderr parsl.dataflow.dflow DEBUG: Task 457 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 457 submitted for App characterizeImage, waiting on task 456 parsl.dataflow.dflow DEBUG: Task 457 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 457 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 458 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 458 submitted for App calibrate, waiting on task 457 parsl.dataflow.dflow DEBUG: Task 458 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 458 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 459 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 459 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 459 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 459 has memoization hash 7d0eb5467c34fe17af7c1d93220cd1da parsl.dataflow.memoization INFO: Task 459 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456cc5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 459 try 0 launched on executor multi with executor id 154 parsl.dataflow.dflow INFO: Standard output for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/9a86b830-df18-4a82-a7bc-8e6c45b30715_isr_855042_19.stdout parsl.dataflow.dflow INFO: Standard error for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/9a86b830-df18-4a82-a7bc-8e6c45b30715_isr_855042_19.stderr parsl.dataflow.dflow DEBUG: Task 460 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 460 submitted for App characterizeImage, waiting on task 459 parsl.dataflow.dflow DEBUG: Task 460 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 460 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 461 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 461 submitted for App calibrate, waiting on task 460 parsl.dataflow.dflow DEBUG: Task 461 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 461 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 462 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 462 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 462 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 462 has memoization hash fbbe581d079e183c7082d653ce6ec6ee parsl.dataflow.memoization INFO: Task 462 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456cf920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 462 try 0 launched on executor multi with executor id 155 parsl.dataflow.dflow INFO: Standard output for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/e87393c0-ec2b-4720-8714-9f22009ada14_isr_855136_40.stdout parsl.dataflow.dflow INFO: Standard error for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/e87393c0-ec2b-4720-8714-9f22009ada14_isr_855136_40.stderr parsl.dataflow.dflow DEBUG: Task 463 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 463 submitted for App characterizeImage, waiting on task 462 parsl.dataflow.dflow DEBUG: Task 463 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 463 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 464 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 464 submitted for App calibrate, waiting on task 463 parsl.dataflow.dflow DEBUG: Task 464 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 464 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 465 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 465 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 465 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 465 has memoization hash ee08b1b6feffb8ffc0203274414d78ca parsl.dataflow.memoization INFO: Task 465 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456cf600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 465 try 0 launched on executor multi with executor id 156 parsl.dataflow.dflow INFO: Standard output for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/baef370b-a0b1-4d4f-9833-5265384370ac_isr_855042_30.stdout parsl.dataflow.dflow INFO: Standard error for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/baef370b-a0b1-4d4f-9833-5265384370ac_isr_855042_30.stderr parsl.dataflow.dflow DEBUG: Task 466 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 466 submitted for App characterizeImage, waiting on task 465 parsl.dataflow.dflow DEBUG: Task 466 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 466 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 467 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 467 submitted for App calibrate, waiting on task 466 parsl.dataflow.dflow DEBUG: Task 467 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 467 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 468 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 468 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 468 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 468 has memoization hash 4150d9fdd9d282ddd72ca4b8ee6ae931 parsl.dataflow.memoization INFO: Task 468 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456cf2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 468 try 0 launched on executor multi with executor id 157 parsl.dataflow.dflow INFO: Standard output for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/192d1b5f-f18b-464d-bb08-a5dcc976e0eb_isr_855044_15.stdout parsl.dataflow.dflow INFO: Standard error for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/192d1b5f-f18b-464d-bb08-a5dcc976e0eb_isr_855044_15.stderr parsl.dataflow.dflow DEBUG: Task 469 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 469 submitted for App characterizeImage, waiting on task 468 parsl.dataflow.dflow DEBUG: Task 469 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 469 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 470 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 470 submitted for App calibrate, waiting on task 469 parsl.dataflow.dflow DEBUG: Task 470 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 470 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 471 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 471 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 471 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 471 has memoization hash 86c0bc490e722d39ba36d83263c7956f parsl.dataflow.memoization INFO: Task 471 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456cefc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 471 try 0 launched on executor multi with executor id 158 parsl.dataflow.dflow INFO: Standard output for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/618dfb0a-8b2f-43b4-966d-b05d2d0b25a2_isr_855042_42.stdout parsl.dataflow.dflow INFO: Standard error for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/618dfb0a-8b2f-43b4-966d-b05d2d0b25a2_isr_855042_42.stderr parsl.dataflow.dflow DEBUG: Task 472 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 472 submitted for App characterizeImage, waiting on task 471 parsl.dataflow.dflow DEBUG: Task 472 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 472 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 473 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 473 submitted for App calibrate, waiting on task 472 parsl.dataflow.dflow DEBUG: Task 473 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 473 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 474 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 474 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 474 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 474 has memoization hash ba1806526ce8899980204ea0e47191ed parsl.dataflow.memoization INFO: Task 474 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456ceca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 474 try 0 launched on executor multi with executor id 159 parsl.dataflow.dflow INFO: Standard output for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/cd7a6ae7-c44f-4254-a205-e852f6035894_isr_855043_14.stdout parsl.dataflow.dflow INFO: Standard error for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/cd7a6ae7-c44f-4254-a205-e852f6035894_isr_855043_14.stderr parsl.dataflow.dflow DEBUG: Task 475 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 475 submitted for App characterizeImage, waiting on task 474 parsl.dataflow.dflow DEBUG: Task 475 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 475 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 476 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 476 submitted for App calibrate, waiting on task 475 parsl.dataflow.dflow DEBUG: Task 476 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 476 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 477 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 477 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 477 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 477 has memoization hash 84c2c42811faf34156d99a901be27619 parsl.dataflow.memoization INFO: Task 477 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456ccae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 477 try 0 launched on executor multi with executor id 160 parsl.dataflow.dflow INFO: Standard output for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/97dd0409-b671-4190-b110-63bc6e6caec2_isr_855043_60.stdout parsl.dataflow.dflow INFO: Standard error for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/97dd0409-b671-4190-b110-63bc6e6caec2_isr_855043_60.stderr parsl.dataflow.dflow DEBUG: Task 478 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 478 submitted for App characterizeImage, waiting on task 477 parsl.dataflow.dflow DEBUG: Task 478 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 478 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 479 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 479 submitted for App calibrate, waiting on task 478 parsl.dataflow.dflow DEBUG: Task 479 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 479 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 480 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 480 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 480 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 480 has memoization hash 80fa9a5fef4fbfc32ee64680a0511e9e parsl.dataflow.memoization INFO: Task 480 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456ccd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 480 try 0 launched on executor multi with executor id 161 parsl.dataflow.dflow INFO: Standard output for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/989f7af6-c3a7-400a-a6f2-f6edd67cdafe_isr_855044_31.stdout parsl.dataflow.dflow INFO: Standard error for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/989f7af6-c3a7-400a-a6f2-f6edd67cdafe_isr_855044_31.stderr parsl.dataflow.dflow DEBUG: Task 481 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 481 submitted for App characterizeImage, waiting on task 480 parsl.dataflow.dflow DEBUG: Task 481 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 481 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 482 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 482 submitted for App calibrate, waiting on task 481 parsl.dataflow.dflow DEBUG: Task 482 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 482 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 483 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 483 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 483 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 483 has memoization hash e42a90fc27e0f70c7caf8141fe646868 parsl.dataflow.memoization INFO: Task 483 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456cd080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 483 try 0 launched on executor multi with executor id 162 parsl.dataflow.dflow INFO: Standard output for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/faa51076-bc80-4e3c-b2a2-9725f2d2615e_isr_855136_27.stdout parsl.dataflow.dflow INFO: Standard error for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/faa51076-bc80-4e3c-b2a2-9725f2d2615e_isr_855136_27.stderr parsl.dataflow.dflow DEBUG: Task 484 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 484 submitted for App characterizeImage, waiting on task 483 parsl.dataflow.dflow DEBUG: Task 484 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 484 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 485 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 485 submitted for App calibrate, waiting on task 484 parsl.dataflow.dflow DEBUG: Task 485 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 485 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 486 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 486 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 486 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 486 has memoization hash bba9b10969f7395c013b44753e6a8316 parsl.dataflow.memoization INFO: Task 486 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456cd1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 486 try 0 launched on executor multi with executor id 163 parsl.dataflow.dflow INFO: Standard output for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/aa9e7fa8-b76e-44e9-9da8-639c3ecb2c0b_isr_855137_35.stdout parsl.dataflow.dflow INFO: Standard error for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/aa9e7fa8-b76e-44e9-9da8-639c3ecb2c0b_isr_855137_35.stderr parsl.dataflow.dflow DEBUG: Task 487 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 487 submitted for App characterizeImage, waiting on task 486 parsl.dataflow.dflow DEBUG: Task 487 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 487 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 488 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 488 submitted for App calibrate, waiting on task 487 parsl.dataflow.dflow DEBUG: Task 488 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 488 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 489 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 489 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 489 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 489 has memoization hash dc766093383995b9c425ad2bca758d67 parsl.dataflow.memoization INFO: Task 489 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456cd4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 489 try 0 launched on executor multi with executor id 164 parsl.dataflow.dflow INFO: Standard output for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/66c0ca35-fa40-4213-a535-9e50d0ce1ad1_isr_855044_25.stdout parsl.dataflow.dflow INFO: Standard error for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/66c0ca35-fa40-4213-a535-9e50d0ce1ad1_isr_855044_25.stderr parsl.dataflow.dflow DEBUG: Task 490 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 490 submitted for App characterizeImage, waiting on task 489 parsl.dataflow.dflow DEBUG: Task 490 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 490 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 491 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 491 submitted for App calibrate, waiting on task 490 parsl.dataflow.dflow DEBUG: Task 491 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 491 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 492 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 492 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 492 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 492 has memoization hash 723f28b6a6c0be48c2c0671abf18f30f parsl.dataflow.memoization INFO: Task 492 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456cd940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 492 try 0 launched on executor multi with executor id 165 parsl.dataflow.dflow INFO: Standard output for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/57941b70-398f-48a2-8696-41ea06152e61_isr_855043_3.stdout parsl.dataflow.dflow INFO: Standard error for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/57941b70-398f-48a2-8696-41ea06152e61_isr_855043_3.stderr parsl.dataflow.dflow DEBUG: Task 493 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 493 submitted for App characterizeImage, waiting on task 492 parsl.dataflow.dflow DEBUG: Task 493 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 493 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 494 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 494 submitted for App calibrate, waiting on task 493 parsl.dataflow.dflow DEBUG: Task 494 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 494 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 495 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 495 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 495 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 495 has memoization hash 0e3eab4937669d57ae85a3c9af52544f parsl.dataflow.memoization INFO: Task 495 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456cdc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 495 try 0 launched on executor multi with executor id 166 parsl.dataflow.dflow INFO: Standard output for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/4c67464e-c01a-47f9-ab83-408b2783cbab_isr_855138_51.stdout parsl.dataflow.dflow INFO: Standard error for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/4c67464e-c01a-47f9-ab83-408b2783cbab_isr_855138_51.stderr parsl.dataflow.dflow DEBUG: Task 496 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 496 submitted for App characterizeImage, waiting on task 495 parsl.dataflow.dflow DEBUG: Task 496 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 496 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 497 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 497 submitted for App calibrate, waiting on task 496 parsl.dataflow.dflow DEBUG: Task 497 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 497 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 498 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 498 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 498 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 498 has memoization hash 970e3f6453d1bb0854e607c598d19854 parsl.dataflow.memoization INFO: Task 498 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456cdee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 498 try 0 launched on executor multi with executor id 167 parsl.dataflow.dflow INFO: Standard output for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/8d8eb2be-3878-4d8e-9c4c-d00d53cbaef6_isr_855137_28.stdout parsl.dataflow.dflow INFO: Standard error for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/8d8eb2be-3878-4d8e-9c4c-d00d53cbaef6_isr_855137_28.stderr parsl.dataflow.dflow DEBUG: Task 499 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 499 submitted for App characterizeImage, waiting on task 498 parsl.dataflow.dflow DEBUG: Task 499 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 499 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 500 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 500 submitted for App calibrate, waiting on task 499 parsl.dataflow.dflow DEBUG: Task 500 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 500 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 501 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 501 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 501 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 501 has memoization hash 7c02d0d6c378c7a180e6d5bb9893f678 parsl.dataflow.memoization INFO: Task 501 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456ce3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 501 try 0 launched on executor multi with executor id 168 parsl.dataflow.dflow INFO: Standard output for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/ce520975-7be5-4fae-8d3d-e996d6e5fdef_isr_855042_13.stdout parsl.dataflow.dflow INFO: Standard error for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/ce520975-7be5-4fae-8d3d-e996d6e5fdef_isr_855042_13.stderr parsl.dataflow.dflow DEBUG: Task 502 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 502 submitted for App characterizeImage, waiting on task 501 parsl.dataflow.dflow DEBUG: Task 502 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 502 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 503 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 503 submitted for App calibrate, waiting on task 502 parsl.dataflow.dflow DEBUG: Task 503 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 503 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 504 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 504 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 504 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 504 has memoization hash fb44dffcc6c4979ac6b7f07f5f46b514 parsl.dataflow.memoization INFO: Task 504 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456ce660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 504 try 0 launched on executor multi with executor id 169 parsl.dataflow.dflow INFO: Standard output for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/b752be77-cc5d-4438-92a1-b783ba5959a5_isr_855042_15.stdout parsl.dataflow.dflow INFO: Standard error for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/b752be77-cc5d-4438-92a1-b783ba5959a5_isr_855042_15.stderr parsl.dataflow.dflow DEBUG: Task 505 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 505 submitted for App characterizeImage, waiting on task 504 parsl.dataflow.dflow DEBUG: Task 505 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 505 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 506 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 506 submitted for App calibrate, waiting on task 505 parsl.dataflow.dflow DEBUG: Task 506 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 506 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 507 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 507 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 507 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 507 has memoization hash 82c627c7a08b63da0d2afe30d9e5e65a parsl.dataflow.memoization INFO: Task 507 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456ce980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 507 try 0 launched on executor multi with executor id 170 parsl.dataflow.dflow INFO: Standard output for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/e3012e20-3404-4d02-9529-18e033866b50_isr_855044_27.stdout parsl.dataflow.dflow INFO: Standard error for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/e3012e20-3404-4d02-9529-18e033866b50_isr_855044_27.stderr parsl.dataflow.dflow DEBUG: Task 508 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 508 submitted for App characterizeImage, waiting on task 507 parsl.dataflow.dflow DEBUG: Task 508 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 508 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 509 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 509 submitted for App calibrate, waiting on task 508 parsl.dataflow.dflow DEBUG: Task 509 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 509 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 510 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 510 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 510 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 510 has memoization hash c5bdba4a1b0ded83afa799fa14d2da3c parsl.dataflow.memoization INFO: Task 510 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456cfce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 510 try 0 launched on executor multi with executor id 171 parsl.dataflow.dflow INFO: Standard output for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/4aced42b-8200-4ec1-957d-425ba34f1d81_isr_855042_44.stdout parsl.dataflow.dflow INFO: Standard error for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/4aced42b-8200-4ec1-957d-425ba34f1d81_isr_855042_44.stderr parsl.dataflow.dflow DEBUG: Task 511 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 511 submitted for App characterizeImage, waiting on task 510 parsl.dataflow.dflow DEBUG: Task 511 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 511 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 512 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 512 submitted for App calibrate, waiting on task 511 parsl.dataflow.dflow DEBUG: Task 512 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 512 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 513 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 513 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 513 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 513 has memoization hash 65ecebbe71d6f27c3c780231de1ce243 parsl.dataflow.memoization INFO: Task 513 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45700040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 513 try 0 launched on executor multi with executor id 172 parsl.dataflow.dflow INFO: Standard output for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/ad9def62-ef36-40e2-89da-49b6780f9c64_isr_855043_41.stdout parsl.dataflow.dflow INFO: Standard error for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/ad9def62-ef36-40e2-89da-49b6780f9c64_isr_855043_41.stderr parsl.dataflow.dflow DEBUG: Task 514 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 514 submitted for App characterizeImage, waiting on task 513 parsl.dataflow.dflow DEBUG: Task 514 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 514 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 515 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 515 submitted for App calibrate, waiting on task 514 parsl.dataflow.dflow DEBUG: Task 515 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 515 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 516 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 516 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 516 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 516 has memoization hash c9b9def846d1a0b1b467ee7a5f50e6a5 parsl.dataflow.memoization INFO: Task 516 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45700360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 516 try 0 launched on executor multi with executor id 173 parsl.dataflow.dflow INFO: Standard output for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/910e0660-ef1e-4b11-8324-de22875c6300_isr_855137_17.stdout parsl.dataflow.dflow INFO: Standard error for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/910e0660-ef1e-4b11-8324-de22875c6300_isr_855137_17.stderr parsl.dataflow.dflow DEBUG: Task 517 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 517 submitted for App characterizeImage, waiting on task 516 parsl.dataflow.dflow DEBUG: Task 517 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 517 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 518 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 518 submitted for App calibrate, waiting on task 517 parsl.dataflow.dflow DEBUG: Task 518 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 518 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 519 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 519 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 519 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 519 has memoization hash af79bae9d64cdb231913ee3c91c40a72 parsl.dataflow.memoization INFO: Task 519 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45700680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-08 07:15:12 proc_lsst.multi:170 [INFO] [multi] job_id 16912284 proc_lsst.multi INFO: [multi] job_id 16912284 2024-03-08 07:15:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-08 07:15:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:12 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16912284 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16912284 parsl.executors.status_handling DEBUG: Launched block 2 on executor multi with job ID 16912284 parsl.executors.status_handling INFO: Allocated block ID 3 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3 2024-03-08 07:15:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 2024-03-08 07:15:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-08 07:15:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 519 try 0 launched on executor multi with executor id 174 parsl.dataflow.dflow INFO: Standard output for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/e4c88f10-6c79-47ce-98c1-88413c62fd06_isr_855137_41.stdout parsl.dataflow.dflow INFO: Standard error for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/e4c88f10-6c79-47ce-98c1-88413c62fd06_isr_855137_41.stderr parsl.dataflow.dflow DEBUG: Task 520 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 520 submitted for App characterizeImage, waiting on task 519 parsl.dataflow.dflow DEBUG: Task 520 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 520 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 521 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 521 submitted for App calibrate, waiting on task 520 parsl.dataflow.dflow DEBUG: Task 521 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 521 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 522 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 522 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 522 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 522 has memoization hash cb197d534e90cca838ac423836bb9def parsl.dataflow.memoization INFO: Task 522 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45700720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 522 try 0 launched on executor multi with executor id 175 parsl.dataflow.dflow INFO: Standard output for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/300a5f14-ac72-4817-8114-77ec84973af4_isr_855137_39.stdout parsl.dataflow.dflow INFO: Standard error for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/300a5f14-ac72-4817-8114-77ec84973af4_isr_855137_39.stderr parsl.dataflow.dflow DEBUG: Task 523 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 523 submitted for App characterizeImage, waiting on task 522 parsl.dataflow.dflow DEBUG: Task 523 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 523 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 524 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 524 submitted for App calibrate, waiting on task 523 parsl.dataflow.dflow DEBUG: Task 524 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 524 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 525 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 525 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 525 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 525 has memoization hash 3726acaa1bb5d5a64f9a03bf3869b914 parsl.dataflow.memoization INFO: Task 525 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45700540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 525 try 0 launched on executor multi with executor id 176 parsl.dataflow.dflow INFO: Standard output for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/ce4acd6b-7b43-407e-bd08-6d9fd331679b_isr_855042_26.stdout parsl.dataflow.dflow INFO: Standard error for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/ce4acd6b-7b43-407e-bd08-6d9fd331679b_isr_855042_26.stderr parsl.dataflow.dflow DEBUG: Task 526 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 526 submitted for App characterizeImage, waiting on task 525 parsl.dataflow.dflow DEBUG: Task 526 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 526 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 527 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 527 submitted for App calibrate, waiting on task 526 parsl.dataflow.dflow DEBUG: Task 527 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 527 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 528 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 528 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 528 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 528 has memoization hash d275d23c5c1cad760a3a20a65aa04adf parsl.dataflow.memoization INFO: Task 528 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45703d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 528 try 0 launched on executor multi with executor id 177 parsl.dataflow.dflow INFO: Standard output for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/7a94c3c8-f91b-4c5c-b69f-c5a932a0f8c6_isr_855044_38.stdout parsl.dataflow.dflow INFO: Standard error for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/7a94c3c8-f91b-4c5c-b69f-c5a932a0f8c6_isr_855044_38.stderr parsl.dataflow.dflow DEBUG: Task 529 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 529 submitted for App characterizeImage, waiting on task 528 parsl.dataflow.dflow DEBUG: Task 529 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 529 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 530 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 530 submitted for App calibrate, waiting on task 529 parsl.dataflow.dflow DEBUG: Task 530 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 530 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 531 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 531 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 531 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 531 has memoization hash 7ba9c617e382ce3da0d2df60698f4087 parsl.dataflow.memoization INFO: Task 531 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45703a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 531 try 0 launched on executor multi with executor id 178 parsl.dataflow.dflow INFO: Standard output for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/dc1bcaac-aeb8-4c8a-bccf-fe1aafe62f41_isr_855042_32.stdout parsl.dataflow.dflow INFO: Standard error for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/dc1bcaac-aeb8-4c8a-bccf-fe1aafe62f41_isr_855042_32.stderr parsl.dataflow.dflow DEBUG: Task 532 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 532 submitted for App characterizeImage, waiting on task 531 parsl.dataflow.dflow DEBUG: Task 532 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 532 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 533 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 533 submitted for App calibrate, waiting on task 532 parsl.dataflow.dflow DEBUG: Task 533 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 533 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 534 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 534 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 534 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 534 has memoization hash 2472f23f2d09bac1191f9ec52387c664 parsl.dataflow.memoization INFO: Task 534 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45703740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 534 try 0 launched on executor multi with executor id 179 parsl.dataflow.dflow INFO: Standard output for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/37fe96d8-e571-4da3-8893-5362b056eea5_isr_855042_7.stdout parsl.dataflow.dflow INFO: Standard error for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/37fe96d8-e571-4da3-8893-5362b056eea5_isr_855042_7.stderr parsl.dataflow.dflow DEBUG: Task 535 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 535 submitted for App characterizeImage, waiting on task 534 parsl.dataflow.dflow DEBUG: Task 535 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 535 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 536 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 536 submitted for App calibrate, waiting on task 535 parsl.dataflow.dflow DEBUG: Task 536 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 536 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 537 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 537 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 537 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 537 has memoization hash 3fc68b7b32af5d5a3d6fe78d466fb170 parsl.dataflow.memoization INFO: Task 537 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45700f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 537 try 0 launched on executor multi with executor id 180 parsl.dataflow.dflow INFO: Standard output for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/529d2866-60c7-4197-a72e-ce3bcb5c2f00_isr_855043_10.stdout parsl.dataflow.dflow INFO: Standard error for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/529d2866-60c7-4197-a72e-ce3bcb5c2f00_isr_855043_10.stderr parsl.dataflow.dflow DEBUG: Task 538 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 538 submitted for App characterizeImage, waiting on task 537 parsl.dataflow.dflow DEBUG: Task 538 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 538 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 539 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 539 submitted for App calibrate, waiting on task 538 parsl.dataflow.dflow DEBUG: Task 539 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 539 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 540 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 540 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 540 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 540 has memoization hash 3d4777dcb2afc7f7724a13b68ae6523a parsl.dataflow.memoization INFO: Task 540 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45701300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 540 try 0 launched on executor multi with executor id 181 parsl.dataflow.dflow INFO: Standard output for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/45b5e960-377b-4f5d-bd3c-2a7f17c220d3_isr_855044_20.stdout parsl.dataflow.dflow INFO: Standard error for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/45b5e960-377b-4f5d-bd3c-2a7f17c220d3_isr_855044_20.stderr parsl.dataflow.dflow DEBUG: Task 541 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 541 submitted for App characterizeImage, waiting on task 540 parsl.dataflow.dflow DEBUG: Task 541 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 541 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 542 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 542 submitted for App calibrate, waiting on task 541 parsl.dataflow.dflow DEBUG: Task 542 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 542 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 543 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 543 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 543 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 543 has memoization hash cf4ff54f0fcc73e45ce05e0e1d76c24c parsl.dataflow.memoization INFO: Task 543 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa457014e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 543 try 0 launched on executor multi with executor id 182 parsl.dataflow.dflow INFO: Standard output for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/e4076bcb-615d-43e1-9dd1-69b7a2af3c8d_isr_855138_18.stdout parsl.dataflow.dflow INFO: Standard error for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/e4076bcb-615d-43e1-9dd1-69b7a2af3c8d_isr_855138_18.stderr parsl.dataflow.dflow DEBUG: Task 544 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 544 submitted for App characterizeImage, waiting on task 543 parsl.dataflow.dflow DEBUG: Task 544 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 544 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 545 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 545 submitted for App calibrate, waiting on task 544 parsl.dataflow.dflow DEBUG: Task 545 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 545 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 546 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 546 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 546 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 546 has memoization hash 3c9d0c0a42440c2619d5a07f54938d9c parsl.dataflow.memoization INFO: Task 546 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa457019e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 546 try 0 launched on executor multi with executor id 183 parsl.dataflow.dflow INFO: Standard output for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/19c510a4-32c9-4c8e-ba61-e11f846fb755_isr_855138_62.stdout parsl.dataflow.dflow INFO: Standard error for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/19c510a4-32c9-4c8e-ba61-e11f846fb755_isr_855138_62.stderr parsl.dataflow.dflow DEBUG: Task 547 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 547 submitted for App characterizeImage, waiting on task 546 parsl.dataflow.dflow DEBUG: Task 547 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 547 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 548 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 548 submitted for App calibrate, waiting on task 547 parsl.dataflow.dflow DEBUG: Task 548 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 548 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 549 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 549 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 549 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 549 has memoization hash afa52808ee6372d1f3261c63526e56df parsl.dataflow.memoization INFO: Task 549 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45701d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 549 try 0 launched on executor multi with executor id 184 parsl.dataflow.dflow INFO: Standard output for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/7e1d558a-fdb4-4f85-822e-6335ed8c9f9b_isr_855044_62.stdout parsl.dataflow.dflow INFO: Standard error for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/7e1d558a-fdb4-4f85-822e-6335ed8c9f9b_isr_855044_62.stderr parsl.dataflow.dflow DEBUG: Task 550 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 550 submitted for App characterizeImage, waiting on task 549 parsl.dataflow.dflow DEBUG: Task 550 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 550 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 551 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 551 submitted for App calibrate, waiting on task 550 parsl.dataflow.dflow DEBUG: Task 551 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 551 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 552 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 552 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 552 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 552 has memoization hash e0968c759e1b520f2988afa078e1de48 parsl.dataflow.memoization INFO: Task 552 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45701f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 552 try 0 launched on executor multi with executor id 185 parsl.dataflow.dflow INFO: Standard output for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/59a70cea-9c6b-4522-a3b9-b2ace1bde2c0_isr_855044_14.stdout parsl.dataflow.dflow INFO: Standard error for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/59a70cea-9c6b-4522-a3b9-b2ace1bde2c0_isr_855044_14.stderr parsl.dataflow.dflow DEBUG: Task 553 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 553 submitted for App characterizeImage, waiting on task 552 parsl.dataflow.dflow DEBUG: Task 553 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 553 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 554 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 554 submitted for App calibrate, waiting on task 553 parsl.dataflow.dflow DEBUG: Task 554 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 554 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 555 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 555 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 555 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 555 has memoization hash 15e86939a75f5a96588e46a78b9026d5 parsl.dataflow.memoization INFO: Task 555 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa457022a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 555 try 0 launched on executor multi with executor id 186 parsl.dataflow.dflow INFO: Standard output for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/ce71a764-566e-4af7-ac27-bb28363fd409_isr_855042_1.stdout parsl.dataflow.dflow INFO: Standard error for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/ce71a764-566e-4af7-ac27-bb28363fd409_isr_855042_1.stderr parsl.dataflow.dflow DEBUG: Task 556 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 556 submitted for App characterizeImage, waiting on task 555 parsl.dataflow.dflow DEBUG: Task 556 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 556 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 557 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 557 submitted for App calibrate, waiting on task 556 parsl.dataflow.dflow DEBUG: Task 557 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 557 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 558 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 558 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 558 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 558 has memoization hash 71367250e0e47ff555f5ab1fdabb1896 parsl.dataflow.memoization INFO: Task 558 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45702660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 558 try 0 launched on executor multi with executor id 187 parsl.dataflow.dflow INFO: Standard output for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/ff8e4113-73e2-4dcc-9479-cc2965049555_isr_855138_50.stdout parsl.dataflow.dflow INFO: Standard error for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/ff8e4113-73e2-4dcc-9479-cc2965049555_isr_855138_50.stderr parsl.dataflow.dflow DEBUG: Task 559 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 559 submitted for App characterizeImage, waiting on task 558 parsl.dataflow.dflow DEBUG: Task 559 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 559 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 560 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 560 submitted for App calibrate, waiting on task 559 parsl.dataflow.dflow DEBUG: Task 560 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 560 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 561 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 561 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 561 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 561 has memoization hash 79d3926b8903f884f5d762ba1215fa8b parsl.dataflow.memoization INFO: Task 561 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45702980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 561 try 0 launched on executor multi with executor id 188 parsl.dataflow.dflow INFO: Standard output for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/2bd76537-b61d-47a2-948d-75eadd2a5edc_isr_855138_16.stdout parsl.dataflow.dflow INFO: Standard error for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/2bd76537-b61d-47a2-948d-75eadd2a5edc_isr_855138_16.stderr parsl.dataflow.dflow DEBUG: Task 562 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 562 submitted for App characterizeImage, waiting on task 561 parsl.dataflow.dflow DEBUG: Task 562 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 562 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 563 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 563 submitted for App calibrate, waiting on task 562 parsl.dataflow.dflow DEBUG: Task 563 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 563 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 564 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 564 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 564 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 564 has memoization hash 3649ec3db779fb2c0592ae892e3a39ad parsl.dataflow.memoization INFO: Task 564 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45702a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 564 try 0 launched on executor multi with executor id 189 parsl.dataflow.dflow INFO: Standard output for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/0a8fdfdb-c28a-4cc4-9c90-866c5a9795e6_isr_855136_7.stdout parsl.dataflow.dflow INFO: Standard error for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/0a8fdfdb-c28a-4cc4-9c90-866c5a9795e6_isr_855136_7.stderr parsl.dataflow.dflow DEBUG: Task 565 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 565 submitted for App characterizeImage, waiting on task 564 parsl.dataflow.dflow DEBUG: Task 565 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 565 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 566 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 566 submitted for App calibrate, waiting on task 565 parsl.dataflow.dflow DEBUG: Task 566 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 566 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 567 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 567 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 567 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 567 has memoization hash 854802138eadbcd548a21c6b87d5f079 parsl.dataflow.memoization INFO: Task 567 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45703420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 567 try 0 launched on executor multi with executor id 190 parsl.dataflow.dflow INFO: Standard output for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/6becd73c-abe1-49ea-bd82-13ea2cc8949e_isr_855043_50.stdout parsl.dataflow.dflow INFO: Standard error for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/6becd73c-abe1-49ea-bd82-13ea2cc8949e_isr_855043_50.stderr parsl.dataflow.dflow DEBUG: Task 568 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 568 submitted for App characterizeImage, waiting on task 567 parsl.dataflow.dflow DEBUG: Task 568 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 568 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 569 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 569 submitted for App calibrate, waiting on task 568 parsl.dataflow.dflow DEBUG: Task 569 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 569 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 570 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 570 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 570 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 570 has memoization hash f0ae18f4d80f49cdffe1acb27bd7c52e parsl.dataflow.memoization INFO: Task 570 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45703100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 570 try 0 launched on executor multi with executor id 191 parsl.dataflow.dflow INFO: Standard output for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/16673fa5-4f22-4d84-82de-aea6d550cfe5_isr_855043_53.stdout parsl.dataflow.dflow INFO: Standard error for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/16673fa5-4f22-4d84-82de-aea6d550cfe5_isr_855043_53.stderr parsl.dataflow.dflow DEBUG: Task 571 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 571 submitted for App characterizeImage, waiting on task 570 parsl.dataflow.dflow DEBUG: Task 571 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 571 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 572 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 572 submitted for App calibrate, waiting on task 571 parsl.dataflow.dflow DEBUG: Task 572 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 572 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 573 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 573 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 573 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 573 has memoization hash 4029a0046027ad0652898adedc140952 parsl.dataflow.memoization INFO: Task 573 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45700b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-08 07:15:12 proc_lsst.multi:170 [INFO] [multi] job_id 16912285 proc_lsst.multi INFO: [multi] job_id 16912285 2024-03-08 07:15:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-08 07:15:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:12 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16912285 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16912285 parsl.executors.status_handling DEBUG: Launched block 3 on executor multi with job ID 16912285 parsl.executors.status_handling INFO: Allocated block ID 4 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4 2024-03-08 07:15:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 2024-03-08 07:15:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-08 07:15:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 573 try 0 launched on executor multi with executor id 192 parsl.dataflow.dflow INFO: Standard output for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/6ba834bb-b634-4c79-a5fa-31efd7af0028_isr_855138_6.stdout parsl.dataflow.dflow INFO: Standard error for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/6ba834bb-b634-4c79-a5fa-31efd7af0028_isr_855138_6.stderr parsl.dataflow.dflow DEBUG: Task 574 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 574 submitted for App characterizeImage, waiting on task 573 parsl.dataflow.dflow DEBUG: Task 574 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 574 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 575 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 575 submitted for App calibrate, waiting on task 574 parsl.dataflow.dflow DEBUG: Task 575 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 575 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 576 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 576 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 576 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 576 has memoization hash f0ea5cd7576695fcc736844aed65e1c3 parsl.dataflow.memoization INFO: Task 576 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45740040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 576 try 0 launched on executor multi with executor id 193 parsl.dataflow.dflow INFO: Standard output for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/3955e772-5af4-4770-b169-e3203ee81886_isr_855044_18.stdout parsl.dataflow.dflow INFO: Standard error for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/3955e772-5af4-4770-b169-e3203ee81886_isr_855044_18.stderr parsl.dataflow.dflow DEBUG: Task 577 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 577 submitted for App characterizeImage, waiting on task 576 parsl.dataflow.dflow DEBUG: Task 577 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 577 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 578 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 578 submitted for App calibrate, waiting on task 577 parsl.dataflow.dflow DEBUG: Task 578 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 578 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 579 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 579 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 579 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 579 has memoization hash eed3514a57245b7e3b51bfd5af55ac0b parsl.dataflow.memoization INFO: Task 579 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45743600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 579 try 0 launched on executor multi with executor id 194 parsl.dataflow.dflow INFO: Standard output for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/87ae1638-4224-40da-b09e-8f406807252e_isr_855043_58.stdout parsl.dataflow.dflow INFO: Standard error for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/87ae1638-4224-40da-b09e-8f406807252e_isr_855043_58.stderr parsl.dataflow.dflow DEBUG: Task 580 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 580 submitted for App characterizeImage, waiting on task 579 parsl.dataflow.dflow DEBUG: Task 580 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 580 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 581 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 581 submitted for App calibrate, waiting on task 580 parsl.dataflow.dflow DEBUG: Task 581 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 581 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 582 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 582 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 582 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 582 has memoization hash 6301c908fb54ecc31ffb69e8d48ab514 parsl.dataflow.memoization INFO: Task 582 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa457432e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 582 try 0 launched on executor multi with executor id 195 parsl.dataflow.dflow INFO: Standard output for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/cb3598df-94bf-4dc0-b56f-1a2bb41f6df0_isr_855042_57.stdout parsl.dataflow.dflow INFO: Standard error for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/cb3598df-94bf-4dc0-b56f-1a2bb41f6df0_isr_855042_57.stderr parsl.dataflow.dflow DEBUG: Task 583 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 583 submitted for App characterizeImage, waiting on task 582 parsl.dataflow.dflow DEBUG: Task 583 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 583 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 584 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 584 submitted for App calibrate, waiting on task 583 parsl.dataflow.dflow DEBUG: Task 584 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 584 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 585 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 585 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 585 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 585 has memoization hash 3a4dba9c467beb17fb486084c66609e6 parsl.dataflow.memoization INFO: Task 585 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45742fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 585 try 0 launched on executor multi with executor id 196 parsl.dataflow.dflow INFO: Standard output for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/ea0bd3f8-9f3f-48df-93aa-87731f39acf1_isr_855042_21.stdout parsl.dataflow.dflow INFO: Standard error for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/ea0bd3f8-9f3f-48df-93aa-87731f39acf1_isr_855042_21.stderr parsl.dataflow.dflow DEBUG: Task 586 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 586 submitted for App characterizeImage, waiting on task 585 parsl.dataflow.dflow DEBUG: Task 586 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 586 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 587 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 587 submitted for App calibrate, waiting on task 586 parsl.dataflow.dflow DEBUG: Task 587 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 587 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 588 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 588 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 588 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 588 has memoization hash d550ea72389fd5af3bc66e24059db2ad parsl.dataflow.memoization INFO: Task 588 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45742ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 588 try 0 launched on executor multi with executor id 197 parsl.dataflow.dflow INFO: Standard output for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/72bf7e26-d949-4292-a8a3-fc02c148777c_isr_855042_48.stdout parsl.dataflow.dflow INFO: Standard error for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/72bf7e26-d949-4292-a8a3-fc02c148777c_isr_855042_48.stderr parsl.dataflow.dflow DEBUG: Task 589 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 589 submitted for App characterizeImage, waiting on task 588 parsl.dataflow.dflow DEBUG: Task 589 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 589 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 590 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 590 submitted for App calibrate, waiting on task 589 parsl.dataflow.dflow DEBUG: Task 590 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 590 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 591 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 591 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 591 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 591 has memoization hash 243bc388a8aa6faff66b92ac85a7ea01 parsl.dataflow.memoization INFO: Task 591 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45742980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 591 try 0 launched on executor multi with executor id 198 parsl.dataflow.dflow INFO: Standard output for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/e801a22d-2413-4ad6-b34d-d75f67ee8544_isr_855138_43.stdout parsl.dataflow.dflow INFO: Standard error for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/e801a22d-2413-4ad6-b34d-d75f67ee8544_isr_855138_43.stderr parsl.dataflow.dflow DEBUG: Task 592 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 592 submitted for App characterizeImage, waiting on task 591 parsl.dataflow.dflow DEBUG: Task 592 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 592 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 593 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 593 submitted for App calibrate, waiting on task 592 parsl.dataflow.dflow DEBUG: Task 593 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 593 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 594 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 594 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 594 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 594 has memoization hash 73655b27bd1312cb08aa01e6acaeab22 parsl.dataflow.memoization INFO: Task 594 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa457407c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 594 try 0 launched on executor multi with executor id 199 parsl.dataflow.dflow INFO: Standard output for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/93c62f2c-782f-4cb7-a9e1-fd456f85e653_isr_855043_2.stdout parsl.dataflow.dflow INFO: Standard error for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/93c62f2c-782f-4cb7-a9e1-fd456f85e653_isr_855043_2.stderr parsl.dataflow.dflow DEBUG: Task 595 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 595 submitted for App characterizeImage, waiting on task 594 parsl.dataflow.dflow DEBUG: Task 595 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 595 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 596 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 596 submitted for App calibrate, waiting on task 595 parsl.dataflow.dflow DEBUG: Task 596 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 596 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 597 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 597 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 597 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 597 has memoization hash 8037ef1571bc80df2e58ed32ff134cc5 parsl.dataflow.memoization INFO: Task 597 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45740ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 597 try 0 launched on executor multi with executor id 200 parsl.dataflow.dflow INFO: Standard output for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/5d9ef8b4-16fe-4181-b436-37fc2b0f9812_isr_855042_33.stdout parsl.dataflow.dflow INFO: Standard error for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/5d9ef8b4-16fe-4181-b436-37fc2b0f9812_isr_855042_33.stderr parsl.dataflow.dflow DEBUG: Task 598 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 598 submitted for App characterizeImage, waiting on task 597 parsl.dataflow.dflow DEBUG: Task 598 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 598 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 599 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 599 submitted for App calibrate, waiting on task 598 parsl.dataflow.dflow DEBUG: Task 599 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 599 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 600 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 600 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 600 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 600 has memoization hash cd9b146dd8c9b3290f4618e331e0a1b1 parsl.dataflow.memoization INFO: Task 600 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45740d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 600 try 0 launched on executor multi with executor id 201 parsl.dataflow.dflow INFO: Standard output for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/9bce308c-adc0-4c79-8b96-ee11f06fa84b_isr_855136_23.stdout parsl.dataflow.dflow INFO: Standard error for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/9bce308c-adc0-4c79-8b96-ee11f06fa84b_isr_855136_23.stderr parsl.dataflow.dflow DEBUG: Task 601 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 601 submitted for App characterizeImage, waiting on task 600 parsl.dataflow.dflow DEBUG: Task 601 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 601 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 602 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 602 submitted for App calibrate, waiting on task 601 parsl.dataflow.dflow DEBUG: Task 602 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 602 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 603 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 603 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 603 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 603 has memoization hash 10a43a1a84d65dfd80214be8df03bcc1 parsl.dataflow.memoization INFO: Task 603 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45741080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 603 try 0 launched on executor multi with executor id 202 parsl.dataflow.dflow INFO: Standard output for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/4c455147-56b0-4fd6-b83a-75276741d57f_isr_855137_23.stdout parsl.dataflow.dflow INFO: Standard error for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/4c455147-56b0-4fd6-b83a-75276741d57f_isr_855137_23.stderr parsl.dataflow.dflow DEBUG: Task 604 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 604 submitted for App characterizeImage, waiting on task 603 parsl.dataflow.dflow DEBUG: Task 604 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 604 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 605 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 605 submitted for App calibrate, waiting on task 604 parsl.dataflow.dflow DEBUG: Task 605 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 605 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 606 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 606 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 606 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 606 has memoization hash 9ff85eb99075a3ab28714a1e2397f4d6 parsl.dataflow.memoization INFO: Task 606 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45741440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 606 try 0 launched on executor multi with executor id 203 parsl.dataflow.dflow INFO: Standard output for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/a5b43f58-d427-49ca-bba3-878b52505dcf_isr_855138_5.stdout parsl.dataflow.dflow INFO: Standard error for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/a5b43f58-d427-49ca-bba3-878b52505dcf_isr_855138_5.stderr parsl.dataflow.dflow DEBUG: Task 607 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 607 submitted for App characterizeImage, waiting on task 606 parsl.dataflow.dflow DEBUG: Task 607 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 607 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 608 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 608 submitted for App calibrate, waiting on task 607 parsl.dataflow.dflow DEBUG: Task 608 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 608 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 609 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 609 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 609 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 609 has memoization hash c7098f097c14a83b9e0bc0b26f6f212b parsl.dataflow.memoization INFO: Task 609 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45741760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 609 try 0 launched on executor multi with executor id 204 parsl.dataflow.dflow INFO: Standard output for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/dc5c83d1-9067-4b29-8fba-e11209a572cc_isr_855044_46.stdout parsl.dataflow.dflow INFO: Standard error for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/dc5c83d1-9067-4b29-8fba-e11209a572cc_isr_855044_46.stderr parsl.dataflow.dflow DEBUG: Task 610 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 610 submitted for App characterizeImage, waiting on task 609 parsl.dataflow.dflow DEBUG: Task 610 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 610 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 611 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 611 submitted for App calibrate, waiting on task 610 parsl.dataflow.dflow DEBUG: Task 611 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 611 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 612 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 612 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 612 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 612 has memoization hash 610488f08e516a17e312499fd33dd092 parsl.dataflow.memoization INFO: Task 612 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45741a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 612 try 0 launched on executor multi with executor id 205 parsl.dataflow.dflow INFO: Standard output for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/dc1bb0d5-3398-49d4-8e5a-3c0722fe0575_isr_855044_29.stdout parsl.dataflow.dflow INFO: Standard error for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/dc1bb0d5-3398-49d4-8e5a-3c0722fe0575_isr_855044_29.stderr parsl.dataflow.dflow DEBUG: Task 613 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 613 submitted for App characterizeImage, waiting on task 612 parsl.dataflow.dflow DEBUG: Task 613 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 613 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 614 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 614 submitted for App calibrate, waiting on task 613 parsl.dataflow.dflow DEBUG: Task 614 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 614 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 615 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 615 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 615 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 615 has memoization hash 3cde280be35757b85eb02f5cd4864edd parsl.dataflow.memoization INFO: Task 615 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45741da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 615 try 0 launched on executor multi with executor id 206 parsl.dataflow.dflow INFO: Standard output for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/9bbe0a88-c4a8-42bc-bc61-78681af41e45_isr_855136_49.stdout parsl.dataflow.dflow INFO: Standard error for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/9bbe0a88-c4a8-42bc-bc61-78681af41e45_isr_855136_49.stderr parsl.dataflow.dflow DEBUG: Task 616 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 616 submitted for App characterizeImage, waiting on task 615 parsl.dataflow.dflow DEBUG: Task 616 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 616 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 617 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 617 submitted for App calibrate, waiting on task 616 parsl.dataflow.dflow DEBUG: Task 617 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 617 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 618 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 618 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 618 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 618 has memoization hash eeeac2118c829491302d72bbc3126ade parsl.dataflow.memoization INFO: Task 618 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa457420c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 618 try 0 launched on executor multi with executor id 207 parsl.dataflow.dflow INFO: Standard output for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/3c45c250-5c01-481f-9886-78c1cb283b48_isr_855136_41.stdout parsl.dataflow.dflow INFO: Standard error for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/3c45c250-5c01-481f-9886-78c1cb283b48_isr_855136_41.stderr parsl.dataflow.dflow DEBUG: Task 619 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 619 submitted for App characterizeImage, waiting on task 618 parsl.dataflow.dflow DEBUG: Task 619 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 619 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 620 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 620 submitted for App calibrate, waiting on task 619 parsl.dataflow.dflow DEBUG: Task 620 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 620 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 621 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 621 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 621 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 621 has memoization hash 875d93fa9ccc8ce0f9c61ebef8878647 parsl.dataflow.memoization INFO: Task 621 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa457423e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 621 try 0 launched on executor multi with executor id 208 parsl.dataflow.dflow INFO: Standard output for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/5ac2c48d-5ab3-41d0-903d-da9dc12c3dbd_isr_855136_12.stdout parsl.dataflow.dflow INFO: Standard error for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/5ac2c48d-5ab3-41d0-903d-da9dc12c3dbd_isr_855136_12.stderr parsl.dataflow.dflow DEBUG: Task 622 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 622 submitted for App characterizeImage, waiting on task 621 parsl.dataflow.dflow DEBUG: Task 622 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 622 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 623 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 623 submitted for App calibrate, waiting on task 622 parsl.dataflow.dflow DEBUG: Task 623 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 623 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 624 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 624 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 624 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 624 has memoization hash b98f5a9400ed73dc0bb225e20943668a parsl.dataflow.memoization INFO: Task 624 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45742660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 624 try 0 launched on executor multi with executor id 209 parsl.dataflow.dflow INFO: Standard output for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/f78f9864-d7dd-40ae-a0d2-4c1596c85ea7_isr_855138_34.stdout parsl.dataflow.dflow INFO: Standard error for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/f78f9864-d7dd-40ae-a0d2-4c1596c85ea7_isr_855138_34.stderr parsl.dataflow.dflow DEBUG: Task 625 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 625 submitted for App characterizeImage, waiting on task 624 parsl.dataflow.dflow DEBUG: Task 625 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 625 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 626 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 626 submitted for App calibrate, waiting on task 625 parsl.dataflow.dflow DEBUG: Task 626 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 626 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 627 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 627 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 627 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 627 has memoization hash 31c55e94be899179bdab687fcfe605d2 parsl.dataflow.memoization INFO: Task 627 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45740180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 627 try 0 launched on executor multi with executor id 210 parsl.dataflow.dflow INFO: Standard output for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/ab2b3952-9747-4552-9306-34b00612399c_isr_855137_60.stdout parsl.dataflow.dflow INFO: Standard error for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/ab2b3952-9747-4552-9306-34b00612399c_isr_855137_60.stderr parsl.dataflow.dflow DEBUG: Task 628 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 628 submitted for App characterizeImage, waiting on task 627 parsl.dataflow.dflow DEBUG: Task 628 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 628 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 629 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 629 submitted for App calibrate, waiting on task 628 parsl.dataflow.dflow DEBUG: Task 629 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 629 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 630 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 630 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 630 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 630 has memoization hash ae3c5bf585a12bf380cf45cde8939a99 parsl.dataflow.memoization INFO: Task 630 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45743a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 630 try 0 launched on executor multi with executor id 211 parsl.dataflow.dflow INFO: Standard output for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/7a047668-758e-468f-9421-637decc85fc1_isr_855044_30.stdout parsl.dataflow.dflow INFO: Standard error for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/7a047668-758e-468f-9421-637decc85fc1_isr_855044_30.stderr parsl.dataflow.dflow DEBUG: Task 631 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 631 submitted for App characterizeImage, waiting on task 630 parsl.dataflow.dflow DEBUG: Task 631 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 631 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 632 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 632 submitted for App calibrate, waiting on task 631 parsl.dataflow.dflow DEBUG: Task 632 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 632 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 633 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 633 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 633 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 633 has memoization hash 40a253af77c35b9e13dd6815fa28b211 parsl.dataflow.memoization INFO: Task 633 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45743d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 633 try 0 launched on executor multi with executor id 212 parsl.dataflow.dflow INFO: Standard output for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/283c6682-faee-4bc8-8224-15ac6c2e3e10_isr_855044_37.stdout parsl.dataflow.dflow INFO: Standard error for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/283c6682-faee-4bc8-8224-15ac6c2e3e10_isr_855044_37.stderr parsl.dataflow.dflow DEBUG: Task 634 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 634 submitted for App characterizeImage, waiting on task 633 parsl.dataflow.dflow DEBUG: Task 634 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 634 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 635 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 635 submitted for App calibrate, waiting on task 634 parsl.dataflow.dflow DEBUG: Task 635 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 635 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 636 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 636 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 636 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 636 has memoization hash d06f8717c2449067698c928fc99a31b7 parsl.dataflow.memoization INFO: Task 636 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558c0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 636 try 0 launched on executor multi with executor id 213 parsl.dataflow.dflow INFO: Standard output for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/be5876af-fddf-4e4f-8c5a-0340b13351b3_isr_855044_50.stdout parsl.dataflow.dflow INFO: Standard error for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/be5876af-fddf-4e4f-8c5a-0340b13351b3_isr_855044_50.stderr parsl.dataflow.dflow DEBUG: Task 637 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 637 submitted for App characterizeImage, waiting on task 636 parsl.dataflow.dflow DEBUG: Task 637 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 637 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 638 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 638 submitted for App calibrate, waiting on task 637 parsl.dataflow.dflow DEBUG: Task 638 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 638 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 639 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 639 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 639 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 639 has memoization hash f91c3ed579234fe73293194a8d0bb040 parsl.dataflow.memoization INFO: Task 639 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-08 07:15:13 proc_lsst.multi:170 [INFO] [multi] job_id 16912286 proc_lsst.multi INFO: [multi] job_id 16912286 2024-03-08 07:15:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-08 07:15:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:13 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16912286 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16912286 parsl.executors.status_handling DEBUG: Launched block 4 on executor multi with job ID 16912286 parsl.executors.status_handling INFO: Allocated block ID 5 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5 2024-03-08 07:15:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 2024-03-08 07:15:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-08 07:15:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 639 try 0 launched on executor multi with executor id 214 parsl.dataflow.dflow INFO: Standard output for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/6b7905cd-744c-483b-a446-6c4d81b9ba6a_isr_855138_47.stdout parsl.dataflow.dflow INFO: Standard error for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/6b7905cd-744c-483b-a446-6c4d81b9ba6a_isr_855138_47.stderr parsl.dataflow.dflow DEBUG: Task 640 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 640 submitted for App characterizeImage, waiting on task 639 parsl.dataflow.dflow DEBUG: Task 640 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 640 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 641 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 641 submitted for App calibrate, waiting on task 640 parsl.dataflow.dflow DEBUG: Task 641 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 641 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 642 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 642 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 642 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 642 has memoization hash b178a77d1c05747f015df355457507e2 parsl.dataflow.memoization INFO: Task 642 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 642 try 0 launched on executor multi with executor id 215 parsl.dataflow.dflow INFO: Standard output for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/69674bdc-bf39-4848-8f3e-c14dd097ba05_isr_855043_33.stdout parsl.dataflow.dflow INFO: Standard error for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/69674bdc-bf39-4848-8f3e-c14dd097ba05_isr_855043_33.stderr parsl.dataflow.dflow DEBUG: Task 643 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 643 submitted for App characterizeImage, waiting on task 642 parsl.dataflow.dflow DEBUG: Task 643 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 643 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 644 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 644 submitted for App calibrate, waiting on task 643 parsl.dataflow.dflow DEBUG: Task 644 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 644 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 645 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 645 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 645 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 645 has memoization hash 176f8e6b8540acb364a73f89de1d2e95 parsl.dataflow.memoization INFO: Task 645 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 645 try 0 launched on executor multi with executor id 216 parsl.dataflow.dflow INFO: Standard output for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/3f6102e3-4418-4a2d-9fd9-1be09135cae9_isr_855043_44.stdout parsl.dataflow.dflow INFO: Standard error for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/3f6102e3-4418-4a2d-9fd9-1be09135cae9_isr_855043_44.stderr parsl.dataflow.dflow DEBUG: Task 646 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 646 submitted for App characterizeImage, waiting on task 645 parsl.dataflow.dflow DEBUG: Task 646 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 646 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 647 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 647 submitted for App calibrate, waiting on task 646 parsl.dataflow.dflow DEBUG: Task 647 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 647 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 648 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 648 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 648 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 648 has memoization hash 33a4d45a6d0ce91cb890a6f40445753c parsl.dataflow.memoization INFO: Task 648 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 648 try 0 launched on executor multi with executor id 217 parsl.dataflow.dflow INFO: Standard output for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/b85d4624-a0b6-4f9b-8cbb-a4b5f1b2375c_isr_855137_26.stdout parsl.dataflow.dflow INFO: Standard error for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/b85d4624-a0b6-4f9b-8cbb-a4b5f1b2375c_isr_855137_26.stderr parsl.dataflow.dflow DEBUG: Task 649 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 649 submitted for App characterizeImage, waiting on task 648 parsl.dataflow.dflow DEBUG: Task 649 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 649 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 650 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 650 submitted for App calibrate, waiting on task 649 parsl.dataflow.dflow DEBUG: Task 650 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 650 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 651 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 651 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 651 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 651 has memoization hash fc4fb17ec2240c520a693d572fe013c9 parsl.dataflow.memoization INFO: Task 651 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 651 try 0 launched on executor multi with executor id 218 parsl.dataflow.dflow INFO: Standard output for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/54fe1f45-3afe-4585-8d7a-1c4cb6d6f838_isr_855042_9.stdout parsl.dataflow.dflow INFO: Standard error for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/54fe1f45-3afe-4585-8d7a-1c4cb6d6f838_isr_855042_9.stderr parsl.dataflow.dflow DEBUG: Task 652 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 652 submitted for App characterizeImage, waiting on task 651 parsl.dataflow.dflow DEBUG: Task 652 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 652 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 653 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 653 submitted for App calibrate, waiting on task 652 parsl.dataflow.dflow DEBUG: Task 653 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 653 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 654 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 654 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 654 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 654 has memoization hash 38a1278a0a896541bcbc2f8d63d75f4c parsl.dataflow.memoization INFO: Task 654 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 654 try 0 launched on executor multi with executor id 219 parsl.dataflow.dflow INFO: Standard output for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/dddc8689-aa67-48f7-8162-5be429c46489_isr_855043_39.stdout parsl.dataflow.dflow INFO: Standard error for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/dddc8689-aa67-48f7-8162-5be429c46489_isr_855043_39.stderr parsl.dataflow.dflow DEBUG: Task 655 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 655 submitted for App characterizeImage, waiting on task 654 parsl.dataflow.dflow DEBUG: Task 655 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 655 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 656 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 656 submitted for App calibrate, waiting on task 655 parsl.dataflow.dflow DEBUG: Task 656 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 656 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 657 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 657 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 657 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 657 has memoization hash d75495a96f491df760efdb8fb622b458 parsl.dataflow.memoization INFO: Task 657 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 657 try 0 launched on executor multi with executor id 220 parsl.dataflow.dflow INFO: Standard output for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/6cc02eb7-13c2-4813-9ee4-30316f3c73ce_isr_855042_36.stdout parsl.dataflow.dflow INFO: Standard error for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/6cc02eb7-13c2-4813-9ee4-30316f3c73ce_isr_855042_36.stderr parsl.dataflow.dflow DEBUG: Task 658 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 658 submitted for App characterizeImage, waiting on task 657 parsl.dataflow.dflow DEBUG: Task 658 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 658 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 659 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 659 submitted for App calibrate, waiting on task 658 parsl.dataflow.dflow DEBUG: Task 659 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 659 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 660 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 660 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 660 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 660 has memoization hash f38aa51b33785206220f7b3ed2ad1533 parsl.dataflow.memoization INFO: Task 660 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 660 try 0 launched on executor multi with executor id 221 parsl.dataflow.dflow INFO: Standard output for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/acfbdb89-4612-41c8-905d-b2c17354061a_isr_855136_47.stdout parsl.dataflow.dflow INFO: Standard error for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/acfbdb89-4612-41c8-905d-b2c17354061a_isr_855136_47.stderr parsl.dataflow.dflow DEBUG: Task 661 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 661 submitted for App characterizeImage, waiting on task 660 parsl.dataflow.dflow DEBUG: Task 661 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 661 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 662 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 662 submitted for App calibrate, waiting on task 661 parsl.dataflow.dflow DEBUG: Task 662 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 662 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 663 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 663 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 663 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 663 has memoization hash e9e3fab5891d2a71b98c09a7aebec164 parsl.dataflow.memoization INFO: Task 663 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 663 try 0 launched on executor multi with executor id 222 parsl.dataflow.dflow INFO: Standard output for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/d73c9bea-e591-48de-8e35-483cae672d65_isr_855042_18.stdout parsl.dataflow.dflow INFO: Standard error for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/d73c9bea-e591-48de-8e35-483cae672d65_isr_855042_18.stderr parsl.dataflow.dflow DEBUG: Task 664 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 664 submitted for App characterizeImage, waiting on task 663 parsl.dataflow.dflow DEBUG: Task 664 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 664 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 665 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 665 submitted for App calibrate, waiting on task 664 parsl.dataflow.dflow DEBUG: Task 665 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 665 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 666 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 666 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 666 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 666 has memoization hash 53e97261018009780ef7d126ed1cb0f5 parsl.dataflow.memoization INFO: Task 666 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 666 try 0 launched on executor multi with executor id 223 parsl.dataflow.dflow INFO: Standard output for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/af10f13f-22b7-49a7-bd72-385a8d7ec726_isr_855043_18.stdout parsl.dataflow.dflow INFO: Standard error for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/af10f13f-22b7-49a7-bd72-385a8d7ec726_isr_855043_18.stderr parsl.dataflow.dflow DEBUG: Task 667 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 667 submitted for App characterizeImage, waiting on task 666 parsl.dataflow.dflow DEBUG: Task 667 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 667 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 668 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 668 submitted for App calibrate, waiting on task 667 parsl.dataflow.dflow DEBUG: Task 668 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 668 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 669 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 669 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 669 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 669 has memoization hash 880a4428a49ef3ce80021ccf44730c1a parsl.dataflow.memoization INFO: Task 669 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 669 try 0 launched on executor multi with executor id 224 parsl.dataflow.dflow INFO: Standard output for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/1a89a24f-2e73-4a5e-ad74-28d4e38ea198_isr_855137_55.stdout parsl.dataflow.dflow INFO: Standard error for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/1a89a24f-2e73-4a5e-ad74-28d4e38ea198_isr_855137_55.stderr parsl.dataflow.dflow DEBUG: Task 670 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 670 submitted for App characterizeImage, waiting on task 669 parsl.dataflow.dflow DEBUG: Task 670 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 670 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 671 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 671 submitted for App calibrate, waiting on task 670 parsl.dataflow.dflow DEBUG: Task 671 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 671 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 672 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 672 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 672 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 672 has memoization hash 1282e725cc7d1feb657f3339c7a5012d parsl.dataflow.memoization INFO: Task 672 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 672 try 0 launched on executor multi with executor id 225 parsl.dataflow.dflow INFO: Standard output for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/fb78e50d-9cfa-455d-9803-184aa0fb616b_isr_855043_16.stdout parsl.dataflow.dflow INFO: Standard error for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/fb78e50d-9cfa-455d-9803-184aa0fb616b_isr_855043_16.stderr parsl.dataflow.dflow DEBUG: Task 673 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 673 submitted for App characterizeImage, waiting on task 672 parsl.dataflow.dflow DEBUG: Task 673 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 673 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 674 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 674 submitted for App calibrate, waiting on task 673 parsl.dataflow.dflow DEBUG: Task 674 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 674 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 675 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 675 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 675 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 675 has memoization hash 429f994fa841117fa57d02bfd851759e parsl.dataflow.memoization INFO: Task 675 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 675 try 0 launched on executor multi with executor id 226 parsl.dataflow.dflow INFO: Standard output for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/3645ad0c-3140-4e83-a056-9a0ab7e0f86e_isr_855044_43.stdout parsl.dataflow.dflow INFO: Standard error for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/3645ad0c-3140-4e83-a056-9a0ab7e0f86e_isr_855044_43.stderr parsl.dataflow.dflow DEBUG: Task 676 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 676 submitted for App characterizeImage, waiting on task 675 parsl.dataflow.dflow DEBUG: Task 676 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 676 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 677 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 677 submitted for App calibrate, waiting on task 676 parsl.dataflow.dflow DEBUG: Task 677 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 677 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 678 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 678 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 678 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 678 has memoization hash 6faffc285b5819da6b033bed64e754ed parsl.dataflow.memoization INFO: Task 678 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 678 try 0 launched on executor multi with executor id 227 parsl.dataflow.dflow INFO: Standard output for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/ea4df522-cb2f-4235-83a4-c1d408b1d2bb_isr_855136_11.stdout parsl.dataflow.dflow INFO: Standard error for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/ea4df522-cb2f-4235-83a4-c1d408b1d2bb_isr_855136_11.stderr parsl.dataflow.dflow DEBUG: Task 679 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 679 submitted for App characterizeImage, waiting on task 678 parsl.dataflow.dflow DEBUG: Task 679 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 679 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 680 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 680 submitted for App calibrate, waiting on task 679 parsl.dataflow.dflow DEBUG: Task 680 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 680 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 681 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 681 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 681 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 681 has memoization hash eaf63182117ade877da40e9f81c96403 parsl.dataflow.memoization INFO: Task 681 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 681 try 0 launched on executor multi with executor id 228 parsl.dataflow.dflow INFO: Standard output for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/61f56ff7-8046-4852-bcff-29e80fdae1f3_isr_855043_28.stdout parsl.dataflow.dflow INFO: Standard error for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/61f56ff7-8046-4852-bcff-29e80fdae1f3_isr_855043_28.stderr parsl.dataflow.dflow DEBUG: Task 682 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 682 submitted for App characterizeImage, waiting on task 681 parsl.dataflow.dflow DEBUG: Task 682 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 682 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 683 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 683 submitted for App calibrate, waiting on task 682 parsl.dataflow.dflow DEBUG: Task 683 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 683 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 684 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 684 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 684 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 684 has memoization hash 463891bf1da200cadda72579d0755442 parsl.dataflow.memoization INFO: Task 684 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 684 try 0 launched on executor multi with executor id 229 parsl.dataflow.dflow INFO: Standard output for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/379d8e0b-f22d-4e8b-9780-920d4b25183d_isr_855044_52.stdout parsl.dataflow.dflow INFO: Standard error for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/379d8e0b-f22d-4e8b-9780-920d4b25183d_isr_855044_52.stderr parsl.dataflow.dflow DEBUG: Task 685 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 685 submitted for App characterizeImage, waiting on task 684 parsl.dataflow.dflow DEBUG: Task 685 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 685 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 686 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 686 submitted for App calibrate, waiting on task 685 parsl.dataflow.dflow DEBUG: Task 686 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 686 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 687 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 687 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 687 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 687 has memoization hash abe1781027a7e6ee598cc46b5fbafb24 parsl.dataflow.memoization INFO: Task 687 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 687 try 0 launched on executor multi with executor id 230 parsl.dataflow.dflow INFO: Standard output for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/cc103e0d-f02b-4611-b54c-30f2558e9bea_isr_855136_15.stdout parsl.dataflow.dflow INFO: Standard error for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/cc103e0d-f02b-4611-b54c-30f2558e9bea_isr_855136_15.stderr parsl.dataflow.dflow DEBUG: Task 688 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 688 submitted for App characterizeImage, waiting on task 687 parsl.dataflow.dflow DEBUG: Task 688 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 688 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 689 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 689 submitted for App calibrate, waiting on task 688 parsl.dataflow.dflow DEBUG: Task 689 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 689 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 690 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 690 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 690 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 690 has memoization hash 7bcfd91e9a270afa16d4c8d07df2ff95 parsl.dataflow.memoization INFO: Task 690 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 690 try 0 launched on executor multi with executor id 231 parsl.dataflow.dflow INFO: Standard output for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/912b3cd1-8e42-40d7-8406-52a4d32bb5ca_isr_855136_20.stdout parsl.dataflow.dflow INFO: Standard error for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/912b3cd1-8e42-40d7-8406-52a4d32bb5ca_isr_855136_20.stderr parsl.dataflow.dflow DEBUG: Task 691 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 691 submitted for App characterizeImage, waiting on task 690 parsl.dataflow.dflow DEBUG: Task 691 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 691 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 692 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 692 submitted for App calibrate, waiting on task 691 parsl.dataflow.dflow DEBUG: Task 692 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 692 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 693 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 693 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 693 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 693 has memoization hash 660ab07cf0ac316b22b01bea1311d8f0 parsl.dataflow.memoization INFO: Task 693 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 693 try 0 launched on executor multi with executor id 232 parsl.dataflow.dflow INFO: Standard output for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/21d1c744-f237-40b4-8f3f-6af59d76f710_isr_855138_29.stdout parsl.dataflow.dflow INFO: Standard error for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/21d1c744-f237-40b4-8f3f-6af59d76f710_isr_855138_29.stderr parsl.dataflow.dflow DEBUG: Task 694 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 694 submitted for App characterizeImage, waiting on task 693 parsl.dataflow.dflow DEBUG: Task 694 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 694 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 695 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 695 submitted for App calibrate, waiting on task 694 parsl.dataflow.dflow DEBUG: Task 695 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 695 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 696 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 696 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 696 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 696 has memoization hash d1ea4ada6592e3ef8d2b5c962bc5561e parsl.dataflow.memoization INFO: Task 696 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 696 try 0 launched on executor multi with executor id 233 parsl.dataflow.dflow INFO: Standard output for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/bc1e90fa-ba39-4716-b3df-42e4a5fdabe0_isr_855137_8.stdout parsl.dataflow.dflow INFO: Standard error for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/bc1e90fa-ba39-4716-b3df-42e4a5fdabe0_isr_855137_8.stderr parsl.dataflow.dflow DEBUG: Task 697 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 697 submitted for App characterizeImage, waiting on task 696 parsl.dataflow.dflow DEBUG: Task 697 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 697 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 698 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 698 submitted for App calibrate, waiting on task 697 parsl.dataflow.dflow DEBUG: Task 698 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 698 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 699 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 699 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 699 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 699 has memoization hash a51831e452af65f72355761e1d43619d parsl.dataflow.memoization INFO: Task 699 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d02c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 699 try 0 launched on executor multi with executor id 234 parsl.dataflow.dflow INFO: Standard output for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/f47ebad5-a541-4173-b3dd-21e2f00260eb_isr_855043_45.stdout parsl.dataflow.dflow INFO: Standard error for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/f47ebad5-a541-4173-b3dd-21e2f00260eb_isr_855043_45.stderr parsl.dataflow.dflow DEBUG: Task 700 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 700 submitted for App characterizeImage, waiting on task 699 parsl.dataflow.dflow DEBUG: Task 700 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 700 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 701 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 701 submitted for App calibrate, waiting on task 700 parsl.dataflow.dflow DEBUG: Task 701 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 701 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 702 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 702 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 702 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 702 has memoization hash 663b05cd5835e9310d6b6472e0e9f1aa parsl.dataflow.memoization INFO: Task 702 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d05e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 702 try 0 launched on executor multi with executor id 235 parsl.dataflow.dflow INFO: Standard output for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/e7b06206-1ed8-4ade-9635-79b1a0527d9d_isr_855137_46.stdout parsl.dataflow.dflow INFO: Standard error for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/e7b06206-1ed8-4ade-9635-79b1a0527d9d_isr_855137_46.stderr parsl.dataflow.dflow DEBUG: Task 703 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 703 submitted for App characterizeImage, waiting on task 702 parsl.dataflow.dflow DEBUG: Task 703 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 703 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 704 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 704 submitted for App calibrate, waiting on task 703 parsl.dataflow.dflow DEBUG: Task 704 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 704 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 705 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 705 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 705 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 705 has memoization hash 7534e52d456b598b21a4944cfc74a5e9 parsl.dataflow.memoization INFO: Task 705 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d0900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 705 try 0 launched on executor multi with executor id 236 parsl.dataflow.dflow INFO: Standard output for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/2fc4feaa-4ff2-4a62-a719-90436ef0d9b0_isr_855138_41.stdout parsl.dataflow.dflow INFO: Standard error for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/2fc4feaa-4ff2-4a62-a719-90436ef0d9b0_isr_855138_41.stderr parsl.dataflow.dflow DEBUG: Task 706 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 706 submitted for App characterizeImage, waiting on task 705 parsl.dataflow.dflow DEBUG: Task 706 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 706 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 707 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 707 submitted for App calibrate, waiting on task 706 parsl.dataflow.dflow DEBUG: Task 707 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 707 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 708 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 708 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 708 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 708 has memoization hash 262e9747133968a1939157ca46e849c5 parsl.dataflow.memoization INFO: Task 708 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d0c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 708 try 0 launched on executor multi with executor id 237 parsl.dataflow.dflow INFO: Standard output for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/148889c1-b5c5-4710-8aad-58e58c04b460_isr_855042_5.stdout parsl.dataflow.dflow INFO: Standard error for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/148889c1-b5c5-4710-8aad-58e58c04b460_isr_855042_5.stderr parsl.dataflow.dflow DEBUG: Task 709 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 709 submitted for App characterizeImage, waiting on task 708 parsl.dataflow.dflow DEBUG: Task 709 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 709 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 710 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 710 submitted for App calibrate, waiting on task 709 parsl.dataflow.dflow DEBUG: Task 710 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 710 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 711 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 711 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 711 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 711 has memoization hash 94f7c6fbccdf9229dc431041e63835d2 parsl.dataflow.memoization INFO: Task 711 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d0f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 711 try 0 launched on executor multi with executor id 238 parsl.dataflow.dflow INFO: Standard output for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/8c4cc0e7-28ae-49d2-a27e-a29cc052d637_isr_855138_27.stdout parsl.dataflow.dflow INFO: Standard error for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/8c4cc0e7-28ae-49d2-a27e-a29cc052d637_isr_855138_27.stderr parsl.dataflow.dflow DEBUG: Task 712 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 712 submitted for App characterizeImage, waiting on task 711 parsl.dataflow.dflow DEBUG: Task 712 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 712 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 713 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 713 submitted for App calibrate, waiting on task 712 parsl.dataflow.dflow DEBUG: Task 713 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 713 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 714 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 714 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 714 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 714 has memoization hash a519d7c96d508fdad4ee7d85f0a6df47 parsl.dataflow.memoization INFO: Task 714 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d1260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-08 07:15:13 proc_lsst.multi:170 [INFO] [multi] job_id 16912287 proc_lsst.multi INFO: [multi] job_id 16912287 2024-03-08 07:15:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:13 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16912287 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16912287 parsl.executors.status_handling DEBUG: Launched block 5 on executor multi with job ID 16912287 parsl.executors.status_handling INFO: Allocated block ID 6 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6 2024-03-08 07:15:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 2024-03-08 07:15:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 714 try 0 launched on executor multi with executor id 239 parsl.dataflow.dflow INFO: Standard output for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/8b827292-dac9-4f4b-b09d-ab57f1190a70_isr_855137_12.stdout parsl.dataflow.dflow INFO: Standard error for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/8b827292-dac9-4f4b-b09d-ab57f1190a70_isr_855137_12.stderr parsl.dataflow.dflow DEBUG: Task 715 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 715 submitted for App characterizeImage, waiting on task 714 parsl.dataflow.dflow DEBUG: Task 715 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 715 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 716 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 716 submitted for App calibrate, waiting on task 715 parsl.dataflow.dflow DEBUG: Task 716 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 716 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 717 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 717 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 717 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 717 has memoization hash 1ee5f350526d455ea294562c977627d3 parsl.dataflow.memoization INFO: Task 717 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d1300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 717 try 0 launched on executor multi with executor id 240 parsl.dataflow.dflow INFO: Standard output for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/1b7ae0d4-a839-4e4c-bb9a-bbb1dc545a89_isr_855042_6.stdout parsl.dataflow.dflow INFO: Standard error for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/1b7ae0d4-a839-4e4c-bb9a-bbb1dc545a89_isr_855042_6.stderr parsl.dataflow.dflow DEBUG: Task 718 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 718 submitted for App characterizeImage, waiting on task 717 parsl.dataflow.dflow DEBUG: Task 718 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 718 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 719 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 719 submitted for App calibrate, waiting on task 718 parsl.dataflow.dflow DEBUG: Task 719 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 719 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 720 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 720 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 720 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 720 has memoization hash f3f265cd67c5e4a0f42b62574473bab0 parsl.dataflow.memoization INFO: Task 720 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d16c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 720 try 0 launched on executor multi with executor id 241 parsl.dataflow.dflow INFO: Standard output for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/241df1ee-03d1-4e39-8ec2-639555fd0321_isr_855044_9.stdout parsl.dataflow.dflow INFO: Standard error for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/241df1ee-03d1-4e39-8ec2-639555fd0321_isr_855044_9.stderr parsl.dataflow.dflow DEBUG: Task 721 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 721 submitted for App characterizeImage, waiting on task 720 parsl.dataflow.dflow DEBUG: Task 721 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 721 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 722 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 722 submitted for App calibrate, waiting on task 721 parsl.dataflow.dflow DEBUG: Task 722 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 722 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 723 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 723 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 723 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 723 has memoization hash 5ef80493ebb41925942dfb9e16254e6c parsl.dataflow.memoization INFO: Task 723 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d19e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 723 try 0 launched on executor multi with executor id 242 parsl.dataflow.dflow INFO: Standard output for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/e232475a-8f16-4236-9f66-c84fc5828b0b_isr_855138_12.stdout parsl.dataflow.dflow INFO: Standard error for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/e232475a-8f16-4236-9f66-c84fc5828b0b_isr_855138_12.stderr parsl.dataflow.dflow DEBUG: Task 724 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 724 submitted for App characterizeImage, waiting on task 723 parsl.dataflow.dflow DEBUG: Task 724 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 724 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 725 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 725 submitted for App calibrate, waiting on task 724 parsl.dataflow.dflow DEBUG: Task 725 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 725 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 726 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 726 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 726 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 726 has memoization hash 89d41e6c553e1b133f5939c3d967f0d2 parsl.dataflow.memoization INFO: Task 726 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d1d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 726 try 0 launched on executor multi with executor id 243 parsl.dataflow.dflow INFO: Standard output for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/fad4bc0f-b2ee-47c0-ad9d-285c4dd09eb5_isr_855042_41.stdout parsl.dataflow.dflow INFO: Standard error for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/fad4bc0f-b2ee-47c0-ad9d-285c4dd09eb5_isr_855042_41.stderr parsl.dataflow.dflow DEBUG: Task 727 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 727 submitted for App characterizeImage, waiting on task 726 parsl.dataflow.dflow DEBUG: Task 727 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 727 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 728 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 728 submitted for App calibrate, waiting on task 727 parsl.dataflow.dflow DEBUG: Task 728 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 728 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 729 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 729 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 729 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 729 has memoization hash 6ea3a159c3b953bfa3a78e8a5c3aff8f parsl.dataflow.memoization INFO: Task 729 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d2020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 729 try 0 launched on executor multi with executor id 244 parsl.dataflow.dflow INFO: Standard output for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/9b9bb813-bc72-48c6-9249-a2938d641ea7_isr_855044_47.stdout parsl.dataflow.dflow INFO: Standard error for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/9b9bb813-bc72-48c6-9249-a2938d641ea7_isr_855044_47.stderr parsl.dataflow.dflow DEBUG: Task 730 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 730 submitted for App characterizeImage, waiting on task 729 parsl.dataflow.dflow DEBUG: Task 730 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 730 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 731 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 731 submitted for App calibrate, waiting on task 730 parsl.dataflow.dflow DEBUG: Task 731 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 731 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 732 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 732 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 732 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 732 has memoization hash 1a650a0d6e4b0577646afa360cc4fb78 parsl.dataflow.memoization INFO: Task 732 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d2340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 732 try 0 launched on executor multi with executor id 245 parsl.dataflow.dflow INFO: Standard output for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/7e4f9d0d-4594-4076-bc1f-0df79901e131_isr_855136_29.stdout parsl.dataflow.dflow INFO: Standard error for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/7e4f9d0d-4594-4076-bc1f-0df79901e131_isr_855136_29.stderr parsl.dataflow.dflow DEBUG: Task 733 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 733 submitted for App characterizeImage, waiting on task 732 parsl.dataflow.dflow DEBUG: Task 733 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 733 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 734 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 734 submitted for App calibrate, waiting on task 733 parsl.dataflow.dflow DEBUG: Task 734 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 734 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 735 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 735 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 735 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 735 has memoization hash 7b3154c70555747d2e9ca482ff7fa8f2 parsl.dataflow.memoization INFO: Task 735 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 735 try 0 launched on executor multi with executor id 246 parsl.dataflow.dflow INFO: Standard output for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/aa33b260-f642-4272-8ad4-9a20ca12f149_isr_855044_6.stdout parsl.dataflow.dflow INFO: Standard error for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/aa33b260-f642-4272-8ad4-9a20ca12f149_isr_855044_6.stderr parsl.dataflow.dflow DEBUG: Task 736 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 736 submitted for App characterizeImage, waiting on task 735 parsl.dataflow.dflow DEBUG: Task 736 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 736 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 737 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 737 submitted for App calibrate, waiting on task 736 parsl.dataflow.dflow DEBUG: Task 737 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 737 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 738 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 738 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 738 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 738 has memoization hash 3ec6f206b3f4a384b73eb6ce9d434703 parsl.dataflow.memoization INFO: Task 738 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 738 try 0 launched on executor multi with executor id 247 parsl.dataflow.dflow INFO: Standard output for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/7232fb5d-3692-4d1f-87de-14b8c4830bf4_isr_855136_50.stdout parsl.dataflow.dflow INFO: Standard error for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/7232fb5d-3692-4d1f-87de-14b8c4830bf4_isr_855136_50.stderr parsl.dataflow.dflow DEBUG: Task 739 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 739 submitted for App characterizeImage, waiting on task 738 parsl.dataflow.dflow DEBUG: Task 739 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 739 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 740 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 740 submitted for App calibrate, waiting on task 739 parsl.dataflow.dflow DEBUG: Task 740 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 740 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 741 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 741 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 741 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 741 has memoization hash b85cf7cfc6c7dbe6ce434153ea30d1a0 parsl.dataflow.memoization INFO: Task 741 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 741 try 0 launched on executor multi with executor id 248 parsl.dataflow.dflow INFO: Standard output for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/42ebec5d-6b90-421d-8a33-1304b4821416_isr_855042_50.stdout parsl.dataflow.dflow INFO: Standard error for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/42ebec5d-6b90-421d-8a33-1304b4821416_isr_855042_50.stderr parsl.dataflow.dflow DEBUG: Task 742 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 742 submitted for App characterizeImage, waiting on task 741 parsl.dataflow.dflow DEBUG: Task 742 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 742 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 743 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 743 submitted for App calibrate, waiting on task 742 parsl.dataflow.dflow DEBUG: Task 743 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 743 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 744 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 744 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 744 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 744 has memoization hash 57651aa18b741db85f7af3509bf5594f parsl.dataflow.memoization INFO: Task 744 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 744 try 0 launched on executor multi with executor id 249 parsl.dataflow.dflow INFO: Standard output for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/38be518b-ab5f-41ca-ab76-5778085f0ebe_isr_855043_15.stdout parsl.dataflow.dflow INFO: Standard error for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/38be518b-ab5f-41ca-ab76-5778085f0ebe_isr_855043_15.stderr parsl.dataflow.dflow DEBUG: Task 745 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 745 submitted for App characterizeImage, waiting on task 744 parsl.dataflow.dflow DEBUG: Task 745 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 745 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 746 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 746 submitted for App calibrate, waiting on task 745 parsl.dataflow.dflow DEBUG: Task 746 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 746 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 747 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 747 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 747 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 747 has memoization hash 86d7074255b197fe6e78d9ca393fb10b parsl.dataflow.memoization INFO: Task 747 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 747 try 0 launched on executor multi with executor id 250 parsl.dataflow.dflow INFO: Standard output for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/cdedcd16-ab0e-4f6d-a5c8-13cc762eb183_isr_855136_31.stdout parsl.dataflow.dflow INFO: Standard error for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/cdedcd16-ab0e-4f6d-a5c8-13cc762eb183_isr_855136_31.stderr parsl.dataflow.dflow DEBUG: Task 748 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 748 submitted for App characterizeImage, waiting on task 747 parsl.dataflow.dflow DEBUG: Task 748 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 748 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 749 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 749 submitted for App calibrate, waiting on task 748 parsl.dataflow.dflow DEBUG: Task 749 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 749 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 750 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 750 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 750 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 750 has memoization hash fc6b34532dfbe0384e30df5aaad7d326 parsl.dataflow.memoization INFO: Task 750 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 750 try 0 launched on executor multi with executor id 251 parsl.dataflow.dflow INFO: Standard output for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/63ce4282-0e9c-4f93-a841-61ff9befa86f_isr_855138_44.stdout parsl.dataflow.dflow INFO: Standard error for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/63ce4282-0e9c-4f93-a841-61ff9befa86f_isr_855138_44.stderr parsl.dataflow.dflow DEBUG: Task 751 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 751 submitted for App characterizeImage, waiting on task 750 parsl.dataflow.dflow DEBUG: Task 751 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 751 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 752 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 752 submitted for App calibrate, waiting on task 751 parsl.dataflow.dflow DEBUG: Task 752 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 752 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 753 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 753 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 753 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 753 has memoization hash d4de956d9df5cc3781e7b459ff7d5d05 parsl.dataflow.memoization INFO: Task 753 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d2a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 753 try 0 launched on executor multi with executor id 252 parsl.dataflow.dflow INFO: Standard output for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/7a29285a-9903-4c78-a08b-ed50e8d9a14d_isr_855042_47.stdout parsl.dataflow.dflow INFO: Standard error for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/7a29285a-9903-4c78-a08b-ed50e8d9a14d_isr_855042_47.stderr parsl.dataflow.dflow DEBUG: Task 754 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 754 submitted for App characterizeImage, waiting on task 753 parsl.dataflow.dflow DEBUG: Task 754 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 754 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 755 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 755 submitted for App calibrate, waiting on task 754 parsl.dataflow.dflow DEBUG: Task 755 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 755 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 756 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 756 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 756 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 756 has memoization hash 8084bc3f8f11c9a85b913ab5e7c6771c parsl.dataflow.memoization INFO: Task 756 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d2de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 756 try 0 launched on executor multi with executor id 253 parsl.dataflow.dflow INFO: Standard output for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/b51a2d01-4820-46b1-9415-c0e7abde6f81_isr_855044_10.stdout parsl.dataflow.dflow INFO: Standard error for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/b51a2d01-4820-46b1-9415-c0e7abde6f81_isr_855044_10.stderr parsl.dataflow.dflow DEBUG: Task 757 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 757 submitted for App characterizeImage, waiting on task 756 parsl.dataflow.dflow DEBUG: Task 757 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 757 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 758 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 758 submitted for App calibrate, waiting on task 757 parsl.dataflow.dflow DEBUG: Task 758 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 758 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 759 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 759 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 759 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 759 has memoization hash 8f68f7fd65b1180ee9d5850e584fcbf5 parsl.dataflow.memoization INFO: Task 759 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d31a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 759 try 0 launched on executor multi with executor id 254 parsl.dataflow.dflow INFO: Standard output for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/120b868d-ca98-4700-8835-578eeb788d39_isr_855043_21.stdout parsl.dataflow.dflow INFO: Standard error for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/120b868d-ca98-4700-8835-578eeb788d39_isr_855043_21.stderr parsl.dataflow.dflow DEBUG: Task 760 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 760 submitted for App characterizeImage, waiting on task 759 parsl.dataflow.dflow DEBUG: Task 760 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 760 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 761 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 761 submitted for App calibrate, waiting on task 760 parsl.dataflow.dflow DEBUG: Task 761 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 761 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 762 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 762 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 762 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 762 has memoization hash c7f03924dc4a2bec888aa6902d162bb2 parsl.dataflow.memoization INFO: Task 762 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d34c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 762 try 0 launched on executor multi with executor id 255 parsl.dataflow.dflow INFO: Standard output for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/6fe451c3-0f6c-482d-8c1c-0d44e9fe0bbe_isr_855137_18.stdout parsl.dataflow.dflow INFO: Standard error for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/6fe451c3-0f6c-482d-8c1c-0d44e9fe0bbe_isr_855137_18.stderr parsl.dataflow.dflow DEBUG: Task 763 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 763 submitted for App characterizeImage, waiting on task 762 parsl.dataflow.dflow DEBUG: Task 763 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 763 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 764 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 764 submitted for App calibrate, waiting on task 763 parsl.dataflow.dflow DEBUG: Task 764 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 764 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 765 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 765 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 765 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 765 has memoization hash 317d47b21d3d1eb013e7d530872078f4 parsl.dataflow.memoization INFO: Task 765 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d37e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 765 try 0 launched on executor multi with executor id 256 parsl.dataflow.dflow INFO: Standard output for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/a8dd069a-1d3d-408b-a849-493c6e4560c8_isr_855137_44.stdout parsl.dataflow.dflow INFO: Standard error for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/a8dd069a-1d3d-408b-a849-493c6e4560c8_isr_855137_44.stderr parsl.dataflow.dflow DEBUG: Task 766 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 766 submitted for App characterizeImage, waiting on task 765 parsl.dataflow.dflow DEBUG: Task 766 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 766 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 767 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 767 submitted for App calibrate, waiting on task 766 parsl.dataflow.dflow DEBUG: Task 767 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 767 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 768 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 768 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 768 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 768 has memoization hash 6eacaf42c3904455cc5bf643ba96f4f3 parsl.dataflow.memoization INFO: Task 768 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d3b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 768 try 0 launched on executor multi with executor id 257 parsl.dataflow.dflow INFO: Standard output for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/41fc44e2-dbaa-482c-aea5-68f5974f63ea_isr_855137_1.stdout parsl.dataflow.dflow INFO: Standard error for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/41fc44e2-dbaa-482c-aea5-68f5974f63ea_isr_855137_1.stderr parsl.dataflow.dflow DEBUG: Task 769 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 769 submitted for App characterizeImage, waiting on task 768 parsl.dataflow.dflow DEBUG: Task 769 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 769 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 770 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 770 submitted for App calibrate, waiting on task 769 parsl.dataflow.dflow DEBUG: Task 770 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 770 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 771 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 771 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 771 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 771 has memoization hash 2ff8a041501a2c99094968c7fe007409 parsl.dataflow.memoization INFO: Task 771 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d3e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 771 try 0 launched on executor multi with executor id 258 parsl.dataflow.dflow INFO: Standard output for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/8e69ed9a-5228-4370-a9e6-67dc732f69ba_isr_855043_37.stdout parsl.dataflow.dflow INFO: Standard error for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/8e69ed9a-5228-4370-a9e6-67dc732f69ba_isr_855043_37.stderr parsl.dataflow.dflow DEBUG: Task 772 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 772 submitted for App characterizeImage, waiting on task 771 parsl.dataflow.dflow DEBUG: Task 772 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 772 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 773 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 773 submitted for App calibrate, waiting on task 772 parsl.dataflow.dflow DEBUG: Task 773 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 773 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 774 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 774 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 774 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 774 has memoization hash 318b107635ab1287259e28eef7ca853e parsl.dataflow.memoization INFO: Task 774 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 774 try 0 launched on executor multi with executor id 259 parsl.dataflow.dflow INFO: Standard output for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/cd2d9445-5c55-467b-a5a3-d860c2a45957_isr_855138_38.stdout parsl.dataflow.dflow INFO: Standard error for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/cd2d9445-5c55-467b-a5a3-d860c2a45957_isr_855138_38.stderr parsl.dataflow.dflow DEBUG: Task 775 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 775 submitted for App characterizeImage, waiting on task 774 parsl.dataflow.dflow DEBUG: Task 775 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 775 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 776 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 776 submitted for App calibrate, waiting on task 775 parsl.dataflow.dflow DEBUG: Task 776 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 776 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 777 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 777 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 777 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 777 has memoization hash 9d3f7edb7c0513ace76896c7bbda44c3 parsl.dataflow.memoization INFO: Task 777 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 777 try 0 launched on executor multi with executor id 260 parsl.dataflow.dflow INFO: Standard output for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/31ad4672-437f-4cd2-ad39-41e9c60266ca_isr_855044_26.stdout parsl.dataflow.dflow INFO: Standard error for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/31ad4672-437f-4cd2-ad39-41e9c60266ca_isr_855044_26.stderr parsl.dataflow.dflow DEBUG: Task 778 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 778 submitted for App characterizeImage, waiting on task 777 parsl.dataflow.dflow DEBUG: Task 778 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 778 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 779 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 779 submitted for App calibrate, waiting on task 778 parsl.dataflow.dflow DEBUG: Task 779 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 779 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 780 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 780 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 780 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 780 has memoization hash 23194d8c567897bf0b63cf0a9ea5bf30 parsl.dataflow.memoization INFO: Task 780 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 780 try 0 launched on executor multi with executor id 261 parsl.dataflow.dflow INFO: Standard output for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/f84ac8cb-acf5-4fe6-81ef-e5962ced860a_isr_855042_22.stdout parsl.dataflow.dflow INFO: Standard error for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/f84ac8cb-acf5-4fe6-81ef-e5962ced860a_isr_855042_22.stderr parsl.dataflow.dflow DEBUG: Task 781 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 781 submitted for App characterizeImage, waiting on task 780 parsl.dataflow.dflow DEBUG: Task 781 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 781 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 782 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 782 submitted for App calibrate, waiting on task 781 parsl.dataflow.dflow DEBUG: Task 782 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 782 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 783 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 783 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 783 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 783 has memoization hash 34f69d6d709ca05a7a49258f22b9c374 parsl.dataflow.memoization INFO: Task 783 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 783 try 0 launched on executor multi with executor id 262 parsl.dataflow.dflow INFO: Standard output for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/11023720-c556-4269-8d7f-0f9d1e523a3a_isr_855043_57.stdout parsl.dataflow.dflow INFO: Standard error for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/11023720-c556-4269-8d7f-0f9d1e523a3a_isr_855043_57.stderr parsl.dataflow.dflow DEBUG: Task 784 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 784 submitted for App characterizeImage, waiting on task 783 parsl.dataflow.dflow DEBUG: Task 784 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 784 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 785 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 785 submitted for App calibrate, waiting on task 784 parsl.dataflow.dflow DEBUG: Task 785 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 785 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 786 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 786 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 786 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 786 has memoization hash cf4c62e494e317df4a1b3f7cbaba6d77 parsl.dataflow.memoization INFO: Task 786 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 786 try 0 launched on executor multi with executor id 263 parsl.dataflow.dflow INFO: Standard output for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/4a85b5ef-9c6d-4dc2-bbf0-23c034f1e76b_isr_855137_50.stdout parsl.dataflow.dflow INFO: Standard error for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/4a85b5ef-9c6d-4dc2-bbf0-23c034f1e76b_isr_855137_50.stderr parsl.dataflow.dflow DEBUG: Task 787 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 787 submitted for App characterizeImage, waiting on task 786 parsl.dataflow.dflow DEBUG: Task 787 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 787 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 788 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 788 submitted for App calibrate, waiting on task 787 parsl.dataflow.dflow DEBUG: Task 788 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 788 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 789 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 789 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 789 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 789 has memoization hash 2d32f27383a5fa94ca6daf2fddd6d3d6 parsl.dataflow.memoization INFO: Task 789 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 789 try 0 launched on executor multi with executor id 264 parsl.dataflow.dflow INFO: Standard output for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/882819b5-6792-469f-9a9f-e3d8878d3080_isr_855044_35.stdout parsl.dataflow.dflow INFO: Standard error for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/882819b5-6792-469f-9a9f-e3d8878d3080_isr_855044_35.stderr parsl.dataflow.dflow DEBUG: Task 790 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 790 submitted for App characterizeImage, waiting on task 789 parsl.dataflow.dflow DEBUG: Task 790 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 790 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 791 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 791 submitted for App calibrate, waiting on task 790 parsl.dataflow.dflow DEBUG: Task 791 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 791 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 792 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 792 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 792 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 792 has memoization hash d95bb808aa5641e716f3018e14c4a6c8 parsl.dataflow.memoization INFO: Task 792 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 792 try 0 launched on executor multi with executor id 265 parsl.dataflow.dflow INFO: Standard output for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/4119a780-6cf2-4e1a-9d28-fdec4737bcc7_isr_855043_49.stdout parsl.dataflow.dflow INFO: Standard error for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/4119a780-6cf2-4e1a-9d28-fdec4737bcc7_isr_855043_49.stderr parsl.dataflow.dflow DEBUG: Task 793 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 793 submitted for App characterizeImage, waiting on task 792 parsl.dataflow.dflow DEBUG: Task 793 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 793 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 794 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 794 submitted for App calibrate, waiting on task 793 parsl.dataflow.dflow DEBUG: Task 794 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 794 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 795 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 795 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 795 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 795 has memoization hash 87f9fe05667b7ad458534905b98dbf30 parsl.dataflow.memoization INFO: Task 795 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 795 try 0 launched on executor multi with executor id 266 parsl.dataflow.dflow INFO: Standard output for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/722b50d0-77a2-446a-b260-fd575fa7623b_isr_855136_39.stdout parsl.dataflow.dflow INFO: Standard error for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/722b50d0-77a2-446a-b260-fd575fa7623b_isr_855136_39.stderr parsl.dataflow.dflow DEBUG: Task 796 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 796 submitted for App characterizeImage, waiting on task 795 parsl.dataflow.dflow DEBUG: Task 796 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 796 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 797 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 797 submitted for App calibrate, waiting on task 796 parsl.dataflow.dflow DEBUG: Task 797 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 797 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 798 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 798 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 798 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 798 has memoization hash 6a2f2d3d556ccfd47516f8e811ad82f8 parsl.dataflow.memoization INFO: Task 798 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 798 try 0 launched on executor multi with executor id 267 parsl.dataflow.dflow INFO: Standard output for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/4ac1edb4-337e-4936-bdf7-977ab087ebe6_isr_855138_35.stdout parsl.dataflow.dflow INFO: Standard error for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/4ac1edb4-337e-4936-bdf7-977ab087ebe6_isr_855138_35.stderr parsl.dataflow.dflow DEBUG: Task 799 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 799 submitted for App characterizeImage, waiting on task 798 parsl.dataflow.dflow DEBUG: Task 799 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 799 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 800 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 800 submitted for App calibrate, waiting on task 799 parsl.dataflow.dflow DEBUG: Task 800 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 800 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 801 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 801 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 801 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 801 has memoization hash 7716a9be9b39b8ebfbe02013f15514b3 parsl.dataflow.memoization INFO: Task 801 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 801 try 0 launched on executor multi with executor id 268 parsl.dataflow.dflow INFO: Standard output for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/0e0d23c0-3d5f-4c6e-920d-9ddd40b6ba3a_isr_855138_2.stdout parsl.dataflow.dflow INFO: Standard error for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/0e0d23c0-3d5f-4c6e-920d-9ddd40b6ba3a_isr_855138_2.stderr parsl.dataflow.dflow DEBUG: Task 802 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 802 submitted for App characterizeImage, waiting on task 801 parsl.dataflow.dflow DEBUG: Task 802 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 802 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 803 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 803 submitted for App calibrate, waiting on task 802 parsl.dataflow.dflow DEBUG: Task 803 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 803 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 804 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 804 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 804 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 804 has memoization hash 9241ad9cb0e256bf0e3e91f8140ffff6 parsl.dataflow.memoization INFO: Task 804 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 804 try 0 launched on executor multi with executor id 269 parsl.dataflow.dflow INFO: Standard output for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/99250545-739d-4d7d-afa9-03cf006699a9_isr_855136_6.stdout parsl.dataflow.dflow INFO: Standard error for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/99250545-739d-4d7d-afa9-03cf006699a9_isr_855136_6.stderr parsl.dataflow.dflow DEBUG: Task 805 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 805 submitted for App characterizeImage, waiting on task 804 parsl.dataflow.dflow DEBUG: Task 805 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 805 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 806 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 806 submitted for App calibrate, waiting on task 805 parsl.dataflow.dflow DEBUG: Task 806 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 806 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 807 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 807 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 807 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 807 has memoization hash dfd1c10a9b1cfdda970ad929b9f47611 parsl.dataflow.memoization INFO: Task 807 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 807 try 0 launched on executor multi with executor id 270 parsl.dataflow.dflow INFO: Standard output for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/615623fa-bcf0-4cad-999b-a7e91b81e038_isr_855137_58.stdout parsl.dataflow.dflow INFO: Standard error for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/615623fa-bcf0-4cad-999b-a7e91b81e038_isr_855137_58.stderr parsl.dataflow.dflow DEBUG: Task 808 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 808 submitted for App characterizeImage, waiting on task 807 parsl.dataflow.dflow DEBUG: Task 808 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 808 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 809 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 809 submitted for App calibrate, waiting on task 808 parsl.dataflow.dflow DEBUG: Task 809 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 809 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 810 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 810 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 810 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 810 has memoization hash e35c028a807fd4fedc4c75273862901b parsl.dataflow.memoization INFO: Task 810 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 810 try 0 launched on executor multi with executor id 271 parsl.dataflow.dflow INFO: Standard output for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/806f3ede-1221-43e1-8f3e-20b0134ab696_isr_855044_36.stdout parsl.dataflow.dflow INFO: Standard error for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/806f3ede-1221-43e1-8f3e-20b0134ab696_isr_855044_36.stderr parsl.dataflow.dflow DEBUG: Task 811 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 811 submitted for App characterizeImage, waiting on task 810 parsl.dataflow.dflow DEBUG: Task 811 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 811 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 812 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 812 submitted for App calibrate, waiting on task 811 parsl.dataflow.dflow DEBUG: Task 812 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 812 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 813 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 813 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 813 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 813 has memoization hash b2f7adf347f358dc10e2d567210e95c8 parsl.dataflow.memoization INFO: Task 813 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 813 try 0 launched on executor multi with executor id 272 parsl.dataflow.dflow INFO: Standard output for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/6e645462-531c-4d78-94b8-8149782b61e3_isr_855043_31.stdout parsl.dataflow.dflow INFO: Standard error for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/6e645462-531c-4d78-94b8-8149782b61e3_isr_855043_31.stderr parsl.dataflow.dflow DEBUG: Task 814 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 814 submitted for App characterizeImage, waiting on task 813 parsl.dataflow.dflow DEBUG: Task 814 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 814 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 815 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 815 submitted for App calibrate, waiting on task 814 parsl.dataflow.dflow DEBUG: Task 815 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 815 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 816 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 816 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 816 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 816 has memoization hash 50452e2a24619a7264d77abf1ad0ca62 parsl.dataflow.memoization INFO: Task 816 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 816 try 0 launched on executor multi with executor id 273 parsl.dataflow.dflow INFO: Standard output for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/a9ae5fca-f851-49ce-885a-f2107c5cb72e_isr_855043_35.stdout parsl.dataflow.dflow INFO: Standard error for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/a9ae5fca-f851-49ce-885a-f2107c5cb72e_isr_855043_35.stderr parsl.dataflow.dflow DEBUG: Task 817 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 817 submitted for App characterizeImage, waiting on task 816 parsl.dataflow.dflow DEBUG: Task 817 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 817 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 818 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 818 submitted for App calibrate, waiting on task 817 parsl.dataflow.dflow DEBUG: Task 818 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 818 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 819 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 819 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 819 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 819 has memoization hash 143f20f71f3d40b916f9f656801d50fe parsl.dataflow.memoization INFO: Task 819 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45664040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 819 try 0 launched on executor multi with executor id 274 parsl.dataflow.dflow INFO: Standard output for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/5567f9c2-5569-4a0f-b82d-c5ba949b953a_isr_855042_58.stdout parsl.dataflow.dflow INFO: Standard error for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/5567f9c2-5569-4a0f-b82d-c5ba949b953a_isr_855042_58.stderr parsl.dataflow.dflow DEBUG: Task 820 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 820 submitted for App characterizeImage, waiting on task 819 parsl.dataflow.dflow DEBUG: Task 820 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 820 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 821 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 821 submitted for App calibrate, waiting on task 820 parsl.dataflow.dflow DEBUG: Task 821 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 821 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 822 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 822 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 822 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 822 has memoization hash 2f78ea92675b27af4541e2c2b7c1ba9a parsl.dataflow.memoization INFO: Task 822 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45664360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 822 try 0 launched on executor multi with executor id 275 parsl.dataflow.dflow INFO: Standard output for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/9af5649c-9507-440d-98ca-a5e0675a153b_isr_855137_38.stdout parsl.dataflow.dflow INFO: Standard error for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/9af5649c-9507-440d-98ca-a5e0675a153b_isr_855137_38.stderr parsl.dataflow.dflow DEBUG: Task 823 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 823 submitted for App characterizeImage, waiting on task 822 parsl.dataflow.dflow DEBUG: Task 823 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 823 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 824 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 824 submitted for App calibrate, waiting on task 823 parsl.dataflow.dflow DEBUG: Task 824 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 824 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 825 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 825 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 825 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 825 has memoization hash 26b4698220c33dc4ae36026eb01708a2 parsl.dataflow.memoization INFO: Task 825 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45664680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 825 try 0 launched on executor multi with executor id 276 parsl.dataflow.dflow INFO: Standard output for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/3f8c4794-6f66-4461-b4d0-b89a94902aba_isr_855044_61.stdout parsl.dataflow.dflow INFO: Standard error for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/3f8c4794-6f66-4461-b4d0-b89a94902aba_isr_855044_61.stderr parsl.dataflow.dflow DEBUG: Task 826 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 826 submitted for App characterizeImage, waiting on task 825 parsl.dataflow.dflow DEBUG: Task 826 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 826 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 827 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 827 submitted for App calibrate, waiting on task 826 parsl.dataflow.dflow DEBUG: Task 827 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 827 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 828 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 828 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 828 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 828 has memoization hash fd15c4eae9fa5834ff7d5d8b59f5dd53 parsl.dataflow.memoization INFO: Task 828 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456649a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 828 try 0 launched on executor multi with executor id 277 parsl.dataflow.dflow INFO: Standard output for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/0621f4b8-cc6d-4e59-b3be-9e1f67b45b0b_isr_855136_5.stdout parsl.dataflow.dflow INFO: Standard error for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/0621f4b8-cc6d-4e59-b3be-9e1f67b45b0b_isr_855136_5.stderr parsl.dataflow.dflow DEBUG: Task 829 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 829 submitted for App characterizeImage, waiting on task 828 parsl.dataflow.dflow DEBUG: Task 829 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 829 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 830 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 830 submitted for App calibrate, waiting on task 829 parsl.dataflow.dflow DEBUG: Task 830 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 830 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 831 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 831 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 831 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 831 has memoization hash 1268c0cce88aef4390f84db7984a3d34 parsl.dataflow.memoization INFO: Task 831 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45664cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 831 try 0 launched on executor multi with executor id 278 parsl.dataflow.dflow INFO: Standard output for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/d8783040-41b6-4d72-a883-88d5d2efb20f_isr_855043_34.stdout parsl.dataflow.dflow INFO: Standard error for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/d8783040-41b6-4d72-a883-88d5d2efb20f_isr_855043_34.stderr parsl.dataflow.dflow DEBUG: Task 832 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 832 submitted for App characterizeImage, waiting on task 831 parsl.dataflow.dflow DEBUG: Task 832 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 832 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 833 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 833 submitted for App calibrate, waiting on task 832 parsl.dataflow.dflow DEBUG: Task 833 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 833 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 834 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 834 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 834 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 834 has memoization hash 1cb4cd7301ae0db8be08b16d116891f8 parsl.dataflow.memoization INFO: Task 834 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45664fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 834 try 0 launched on executor multi with executor id 279 parsl.dataflow.dflow INFO: Standard output for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/14da00d1-4641-4a8f-b551-4246626f295e_isr_855044_42.stdout parsl.dataflow.dflow INFO: Standard error for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/14da00d1-4641-4a8f-b551-4246626f295e_isr_855044_42.stderr parsl.dataflow.dflow DEBUG: Task 835 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 835 submitted for App characterizeImage, waiting on task 834 parsl.dataflow.dflow DEBUG: Task 835 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 835 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 836 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 836 submitted for App calibrate, waiting on task 835 parsl.dataflow.dflow DEBUG: Task 836 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 836 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 837 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 837 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 837 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 837 has memoization hash 402c36aaa254565403a076e792eac65c parsl.dataflow.memoization INFO: Task 837 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45665300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-08 07:15:13 proc_lsst.multi:170 [INFO] [multi] job_id 16912288 proc_lsst.multi INFO: [multi] job_id 16912288 2024-03-08 07:15:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-08 07:15:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:13 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16912288 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16912288 parsl.executors.status_handling DEBUG: Launched block 6 on executor multi with job ID 16912288 parsl.executors.status_handling INFO: Allocated block ID 7 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-7 2024-03-08 07:15:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 2024-03-08 07:15:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-08 07:15:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 837 try 0 launched on executor multi with executor id 280 parsl.dataflow.dflow INFO: Standard output for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/e9fee100-7b4d-42db-9652-3b03eee83ecd_isr_855136_9.stdout parsl.dataflow.dflow INFO: Standard error for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/e9fee100-7b4d-42db-9652-3b03eee83ecd_isr_855136_9.stderr parsl.dataflow.dflow DEBUG: Task 838 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 838 submitted for App characterizeImage, waiting on task 837 parsl.dataflow.dflow DEBUG: Task 838 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 838 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 839 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 839 submitted for App calibrate, waiting on task 838 parsl.dataflow.dflow DEBUG: Task 839 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 839 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 840 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 840 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 840 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 840 has memoization hash ec7b16a91b0a77b6d966330d8e4e9f38 parsl.dataflow.memoization INFO: Task 840 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456653a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 840 try 0 launched on executor multi with executor id 281 parsl.dataflow.dflow INFO: Standard output for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/6c450119-a166-4e58-a5b5-f93d7469b309_isr_855137_42.stdout parsl.dataflow.dflow INFO: Standard error for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/6c450119-a166-4e58-a5b5-f93d7469b309_isr_855137_42.stderr parsl.dataflow.dflow DEBUG: Task 841 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 841 submitted for App characterizeImage, waiting on task 840 parsl.dataflow.dflow DEBUG: Task 841 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 841 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 842 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 842 submitted for App calibrate, waiting on task 841 parsl.dataflow.dflow DEBUG: Task 842 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 842 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 843 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 843 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 843 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 843 has memoization hash c91858a65e5c1482545bd457eb634fa6 parsl.dataflow.memoization INFO: Task 843 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45665760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 843 try 0 launched on executor multi with executor id 282 parsl.dataflow.dflow INFO: Standard output for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/aa0a4608-0bdd-44e0-a8e0-9ef5915216ba_isr_855043_40.stdout parsl.dataflow.dflow INFO: Standard error for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/aa0a4608-0bdd-44e0-a8e0-9ef5915216ba_isr_855043_40.stderr parsl.dataflow.dflow DEBUG: Task 844 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 844 submitted for App characterizeImage, waiting on task 843 parsl.dataflow.dflow DEBUG: Task 844 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 844 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 845 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 845 submitted for App calibrate, waiting on task 844 parsl.dataflow.dflow DEBUG: Task 845 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 845 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 846 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 846 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 846 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 846 has memoization hash 989fcd824a4feb6d8aba5c88191a2918 parsl.dataflow.memoization INFO: Task 846 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45665a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 846 try 0 launched on executor multi with executor id 283 parsl.dataflow.dflow INFO: Standard output for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/873081f8-4899-4bc0-b4d5-e2df66f72fcb_isr_855044_7.stdout parsl.dataflow.dflow INFO: Standard error for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/873081f8-4899-4bc0-b4d5-e2df66f72fcb_isr_855044_7.stderr parsl.dataflow.dflow DEBUG: Task 847 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 847 submitted for App characterizeImage, waiting on task 846 parsl.dataflow.dflow DEBUG: Task 847 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 847 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 848 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 848 submitted for App calibrate, waiting on task 847 parsl.dataflow.dflow DEBUG: Task 848 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 848 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 849 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 849 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 849 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 849 has memoization hash 69f5a8142b00f6116d1a9c4b9fe4180d parsl.dataflow.memoization INFO: Task 849 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45665da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 849 try 0 launched on executor multi with executor id 284 parsl.dataflow.dflow INFO: Standard output for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/1e885629-62a6-49a5-aa9c-a76084bd40cb_isr_855042_54.stdout parsl.dataflow.dflow INFO: Standard error for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/1e885629-62a6-49a5-aa9c-a76084bd40cb_isr_855042_54.stderr parsl.dataflow.dflow DEBUG: Task 850 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 850 submitted for App characterizeImage, waiting on task 849 parsl.dataflow.dflow DEBUG: Task 850 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 850 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 851 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 851 submitted for App calibrate, waiting on task 850 parsl.dataflow.dflow DEBUG: Task 851 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 851 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 852 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 852 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 852 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 852 has memoization hash 9de265bcc152cdf1a091ab026546ce5a parsl.dataflow.memoization INFO: Task 852 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456660c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 852 try 0 launched on executor multi with executor id 285 parsl.dataflow.dflow INFO: Standard output for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/879bea37-5690-4edb-b385-5563195f5709_isr_855137_16.stdout parsl.dataflow.dflow INFO: Standard error for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/879bea37-5690-4edb-b385-5563195f5709_isr_855137_16.stderr parsl.dataflow.dflow DEBUG: Task 853 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 853 submitted for App characterizeImage, waiting on task 852 parsl.dataflow.dflow DEBUG: Task 853 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 853 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 854 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 854 submitted for App calibrate, waiting on task 853 parsl.dataflow.dflow DEBUG: Task 854 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 854 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 855 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 855 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 855 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 855 has memoization hash f031644239bbf45549011df31eb89e6b parsl.dataflow.memoization INFO: Task 855 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45499620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 855 try 0 launched on executor multi with executor id 286 parsl.dataflow.dflow INFO: Standard output for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/3de24388-ba3d-4c94-ae9d-3fb9c11f315c_isr_855043_43.stdout parsl.dataflow.dflow INFO: Standard error for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/3de24388-ba3d-4c94-ae9d-3fb9c11f315c_isr_855043_43.stderr parsl.dataflow.dflow DEBUG: Task 856 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 856 submitted for App characterizeImage, waiting on task 855 parsl.dataflow.dflow DEBUG: Task 856 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 856 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 857 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 857 submitted for App calibrate, waiting on task 856 parsl.dataflow.dflow DEBUG: Task 857 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 857 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 858 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 858 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 858 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 858 has memoization hash 2dade082ec4e7888065f28d5e3b7fc94 parsl.dataflow.memoization INFO: Task 858 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45499260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 858 try 0 launched on executor multi with executor id 287 parsl.dataflow.dflow INFO: Standard output for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/cbc1c5ca-cde9-4e4c-b81c-87e92def137f_isr_855136_32.stdout parsl.dataflow.dflow INFO: Standard error for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/cbc1c5ca-cde9-4e4c-b81c-87e92def137f_isr_855136_32.stderr parsl.dataflow.dflow DEBUG: Task 859 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 859 submitted for App characterizeImage, waiting on task 858 parsl.dataflow.dflow DEBUG: Task 859 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 859 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 860 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 860 submitted for App calibrate, waiting on task 859 parsl.dataflow.dflow DEBUG: Task 860 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 860 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 861 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 861 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 861 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 861 has memoization hash ab53e4d6aec468b0150a4c19b2e084e9 parsl.dataflow.memoization INFO: Task 861 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45498f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 861 try 0 launched on executor multi with executor id 288 parsl.dataflow.dflow INFO: Standard output for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/29485926-e381-444d-963b-f3d1d782bba3_isr_855137_53.stdout parsl.dataflow.dflow INFO: Standard error for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/29485926-e381-444d-963b-f3d1d782bba3_isr_855137_53.stderr parsl.dataflow.dflow DEBUG: Task 862 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 862 submitted for App characterizeImage, waiting on task 861 parsl.dataflow.dflow DEBUG: Task 862 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 862 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 863 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 863 submitted for App calibrate, waiting on task 862 parsl.dataflow.dflow DEBUG: Task 863 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 863 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 864 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 864 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 864 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 864 has memoization hash 03fcaede4957643efe11ef6566b27d60 parsl.dataflow.memoization INFO: Task 864 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45498c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 864 try 0 launched on executor multi with executor id 289 parsl.dataflow.dflow INFO: Standard output for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/57bd1a28-f107-4409-bf78-61588090f5bd_isr_855136_61.stdout parsl.dataflow.dflow INFO: Standard error for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/57bd1a28-f107-4409-bf78-61588090f5bd_isr_855136_61.stderr parsl.dataflow.dflow DEBUG: Task 865 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 865 submitted for App characterizeImage, waiting on task 864 parsl.dataflow.dflow DEBUG: Task 865 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 865 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 866 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 866 submitted for App calibrate, waiting on task 865 parsl.dataflow.dflow DEBUG: Task 866 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 866 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 867 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 867 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 867 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 867 has memoization hash 99e20afc4f89beed087c8d891f31c334 parsl.dataflow.memoization INFO: Task 867 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456668e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 867 try 0 launched on executor multi with executor id 290 parsl.dataflow.dflow INFO: Standard output for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/4fcbab09-6512-414b-8612-275243d823b0_isr_855044_13.stdout parsl.dataflow.dflow INFO: Standard error for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/4fcbab09-6512-414b-8612-275243d823b0_isr_855044_13.stderr parsl.dataflow.dflow DEBUG: Task 868 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 868 submitted for App characterizeImage, waiting on task 867 parsl.dataflow.dflow DEBUG: Task 868 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 868 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 869 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 869 submitted for App calibrate, waiting on task 868 parsl.dataflow.dflow DEBUG: Task 869 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 869 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 870 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 870 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 870 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 870 has memoization hash 0d1d29eb43e10b99d7011c874e437cba parsl.dataflow.memoization INFO: Task 870 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45666c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 870 try 0 launched on executor multi with executor id 291 parsl.dataflow.dflow INFO: Standard output for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/0a0c6b1d-258a-4722-a8a0-61dd20994512_isr_855043_20.stdout parsl.dataflow.dflow INFO: Standard error for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/0a0c6b1d-258a-4722-a8a0-61dd20994512_isr_855043_20.stderr parsl.dataflow.dflow DEBUG: Task 871 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 871 submitted for App characterizeImage, waiting on task 870 parsl.dataflow.dflow DEBUG: Task 871 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 871 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 872 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 872 submitted for App calibrate, waiting on task 871 parsl.dataflow.dflow DEBUG: Task 872 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 872 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 873 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 873 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 873 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 873 has memoization hash 91f6627379513df09e6de29e418cf7da parsl.dataflow.memoization INFO: Task 873 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45666d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 873 try 0 launched on executor multi with executor id 292 parsl.dataflow.dflow INFO: Standard output for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/7e4d5ace-abc8-4172-9948-f20b706a5277_isr_855136_2.stdout parsl.dataflow.dflow INFO: Standard error for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/7e4d5ace-abc8-4172-9948-f20b706a5277_isr_855136_2.stderr parsl.dataflow.dflow DEBUG: Task 874 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 874 submitted for App characterizeImage, waiting on task 873 parsl.dataflow.dflow DEBUG: Task 874 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 874 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 875 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 875 submitted for App calibrate, waiting on task 874 parsl.dataflow.dflow DEBUG: Task 875 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 875 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 876 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 876 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 876 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 876 has memoization hash a52fd02b7f4571814f891fd25fcd34cc parsl.dataflow.memoization INFO: Task 876 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45667060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 876 try 0 launched on executor multi with executor id 293 parsl.dataflow.dflow INFO: Standard output for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/2783cd13-a407-4283-8c1a-3636ce90da0d_isr_855137_40.stdout parsl.dataflow.dflow INFO: Standard error for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/2783cd13-a407-4283-8c1a-3636ce90da0d_isr_855137_40.stderr parsl.dataflow.dflow DEBUG: Task 877 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 877 submitted for App characterizeImage, waiting on task 876 parsl.dataflow.dflow DEBUG: Task 877 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 877 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 878 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 878 submitted for App calibrate, waiting on task 877 parsl.dataflow.dflow DEBUG: Task 878 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 878 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 879 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 879 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 879 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 879 has memoization hash c9ee1ec4be37e55875a425c48222133a parsl.dataflow.memoization INFO: Task 879 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456674c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 879 try 0 launched on executor multi with executor id 294 parsl.dataflow.dflow INFO: Standard output for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/9fc2c6c5-371b-4ccf-bb28-964d587c178f_isr_855043_46.stdout parsl.dataflow.dflow INFO: Standard error for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/9fc2c6c5-371b-4ccf-bb28-964d587c178f_isr_855043_46.stderr parsl.dataflow.dflow DEBUG: Task 880 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 880 submitted for App characterizeImage, waiting on task 879 parsl.dataflow.dflow DEBUG: Task 880 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 880 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 881 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 881 submitted for App calibrate, waiting on task 880 parsl.dataflow.dflow DEBUG: Task 881 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 881 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 882 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 882 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 882 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 882 has memoization hash 3f835d2c0cc76cb9a5db89377e4ece3e parsl.dataflow.memoization INFO: Task 882 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456677e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 882 try 0 launched on executor multi with executor id 295 parsl.dataflow.dflow INFO: Standard output for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/d0d7963a-91ce-453c-9381-d623d388ac22_isr_855138_3.stdout parsl.dataflow.dflow INFO: Standard error for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/d0d7963a-91ce-453c-9381-d623d388ac22_isr_855138_3.stderr parsl.dataflow.dflow DEBUG: Task 883 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 883 submitted for App characterizeImage, waiting on task 882 parsl.dataflow.dflow DEBUG: Task 883 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 883 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 884 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 884 submitted for App calibrate, waiting on task 883 parsl.dataflow.dflow DEBUG: Task 884 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 884 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 885 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 885 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 885 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 885 has memoization hash fc376446125c0945b9e0b34937027d35 parsl.dataflow.memoization INFO: Task 885 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45667a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 885 try 0 launched on executor multi with executor id 296 parsl.dataflow.dflow INFO: Standard output for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/d00f4cea-261f-485a-b366-f91e72175331_isr_855042_53.stdout parsl.dataflow.dflow INFO: Standard error for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/d00f4cea-261f-485a-b366-f91e72175331_isr_855042_53.stderr parsl.dataflow.dflow DEBUG: Task 886 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 886 submitted for App characterizeImage, waiting on task 885 parsl.dataflow.dflow DEBUG: Task 886 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 886 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 887 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 887 submitted for App calibrate, waiting on task 886 parsl.dataflow.dflow DEBUG: Task 887 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 887 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 888 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 888 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 888 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 888 has memoization hash 0961c5493f00c7f0579e64d893839a7f parsl.dataflow.memoization INFO: Task 888 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45667f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 888 try 0 launched on executor multi with executor id 297 parsl.dataflow.dflow INFO: Standard output for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/2d64df3f-9441-42e0-94d1-56c0c1218f2e_isr_855043_52.stdout parsl.dataflow.dflow INFO: Standard error for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/2d64df3f-9441-42e0-94d1-56c0c1218f2e_isr_855043_52.stderr parsl.dataflow.dflow DEBUG: Task 889 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 889 submitted for App characterizeImage, waiting on task 888 parsl.dataflow.dflow DEBUG: Task 889 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 889 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 890 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 890 submitted for App calibrate, waiting on task 889 parsl.dataflow.dflow DEBUG: Task 890 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 890 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 891 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 891 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 891 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 891 has memoization hash 96bdd4417c529a8cae27808193209bbc parsl.dataflow.memoization INFO: Task 891 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45498cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 891 try 0 launched on executor multi with executor id 298 parsl.dataflow.dflow INFO: Standard output for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/71a522e0-f337-47d7-ac15-eaefe701b767_isr_855138_40.stdout parsl.dataflow.dflow INFO: Standard error for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/71a522e0-f337-47d7-ac15-eaefe701b767_isr_855138_40.stderr parsl.dataflow.dflow DEBUG: Task 892 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 892 submitted for App characterizeImage, waiting on task 891 parsl.dataflow.dflow DEBUG: Task 892 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 892 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 893 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 893 submitted for App calibrate, waiting on task 892 parsl.dataflow.dflow DEBUG: Task 893 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 893 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 894 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 894 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 894 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 894 has memoization hash 78695601c8570e5983d7a09619f06d22 parsl.dataflow.memoization INFO: Task 894 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454980e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 894 try 0 launched on executor multi with executor id 299 parsl.dataflow.dflow INFO: Standard output for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/0a1bd9fb-13de-46da-8993-abcc822b1772_isr_855042_25.stdout parsl.dataflow.dflow INFO: Standard error for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/0a1bd9fb-13de-46da-8993-abcc822b1772_isr_855042_25.stderr parsl.dataflow.dflow DEBUG: Task 895 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 895 submitted for App characterizeImage, waiting on task 894 parsl.dataflow.dflow DEBUG: Task 895 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 895 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 896 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 896 submitted for App calibrate, waiting on task 895 parsl.dataflow.dflow DEBUG: Task 896 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 896 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 897 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 897 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 897 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 897 has memoization hash 38b5538eec4b496d4fcf3c980a280190 parsl.dataflow.memoization INFO: Task 897 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45498900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 897 try 0 launched on executor multi with executor id 300 parsl.dataflow.dflow INFO: Standard output for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/2bc73e82-e044-452c-89a5-709f2333e4e1_isr_855136_52.stdout parsl.dataflow.dflow INFO: Standard error for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/2bc73e82-e044-452c-89a5-709f2333e4e1_isr_855136_52.stderr parsl.dataflow.dflow DEBUG: Task 898 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 898 submitted for App characterizeImage, waiting on task 897 parsl.dataflow.dflow DEBUG: Task 898 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 898 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 899 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 899 submitted for App calibrate, waiting on task 898 parsl.dataflow.dflow DEBUG: Task 899 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 899 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 900 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 900 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 900 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 900 has memoization hash 373d0aca41ae3931b550ea23462ffcd8 parsl.dataflow.memoization INFO: Task 900 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454985e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 900 try 0 launched on executor multi with executor id 301 parsl.dataflow.dflow INFO: Standard output for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/94543380-79d2-454d-b5a6-cf95c1fa33d0_isr_855138_39.stdout parsl.dataflow.dflow INFO: Standard error for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/94543380-79d2-454d-b5a6-cf95c1fa33d0_isr_855138_39.stderr parsl.dataflow.dflow DEBUG: Task 901 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 901 submitted for App characterizeImage, waiting on task 900 parsl.dataflow.dflow DEBUG: Task 901 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 901 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 902 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 902 submitted for App calibrate, waiting on task 901 parsl.dataflow.dflow DEBUG: Task 902 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 902 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 903 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 903 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 903 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 903 has memoization hash b4c0d5d9388603ab8993868653e38fd7 parsl.dataflow.memoization INFO: Task 903 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45499800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 903 try 0 launched on executor multi with executor id 302 parsl.dataflow.dflow INFO: Standard output for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/633568b0-a778-416a-826b-7c4d54a70afe_isr_855043_56.stdout parsl.dataflow.dflow INFO: Standard error for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/633568b0-a778-416a-826b-7c4d54a70afe_isr_855043_56.stderr parsl.dataflow.dflow DEBUG: Task 904 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 904 submitted for App characterizeImage, waiting on task 903 parsl.dataflow.dflow DEBUG: Task 904 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 904 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 905 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 905 submitted for App calibrate, waiting on task 904 parsl.dataflow.dflow DEBUG: Task 905 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 905 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 906 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 906 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 906 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 906 has memoization hash 874f50b6e0e3dd71bd27e3addb540ce5 parsl.dataflow.memoization INFO: Task 906 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45499b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 906 try 0 launched on executor multi with executor id 303 parsl.dataflow.dflow INFO: Standard output for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/4f01a72b-df1c-41f1-becd-7b90de6136c9_isr_855042_8.stdout parsl.dataflow.dflow INFO: Standard error for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/4f01a72b-df1c-41f1-becd-7b90de6136c9_isr_855042_8.stderr parsl.dataflow.dflow DEBUG: Task 907 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 907 submitted for App characterizeImage, waiting on task 906 parsl.dataflow.dflow DEBUG: Task 907 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 907 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 908 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 908 submitted for App calibrate, waiting on task 907 parsl.dataflow.dflow DEBUG: Task 908 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 908 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 909 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 909 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 909 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 909 has memoization hash 677456621c830ea7532489a00a155a4d parsl.dataflow.memoization INFO: Task 909 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45499e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 909 try 0 launched on executor multi with executor id 304 parsl.dataflow.dflow INFO: Standard output for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/64f55006-bd82-473f-9d09-c172fc733c5e_isr_855043_8.stdout parsl.dataflow.dflow INFO: Standard error for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/64f55006-bd82-473f-9d09-c172fc733c5e_isr_855043_8.stderr parsl.dataflow.dflow DEBUG: Task 910 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 910 submitted for App characterizeImage, waiting on task 909 parsl.dataflow.dflow DEBUG: Task 910 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 910 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 911 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 911 submitted for App calibrate, waiting on task 910 parsl.dataflow.dflow DEBUG: Task 911 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 911 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 912 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 912 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 912 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 912 has memoization hash 35bfc25510fd747924802b73766f8329 parsl.dataflow.memoization INFO: Task 912 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4549a160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 912 try 0 launched on executor multi with executor id 305 parsl.dataflow.dflow INFO: Standard output for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/cd950b6b-63a1-47f7-b354-206913b10685_isr_855042_24.stdout parsl.dataflow.dflow INFO: Standard error for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/cd950b6b-63a1-47f7-b354-206913b10685_isr_855042_24.stderr parsl.dataflow.dflow DEBUG: Task 913 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 913 submitted for App characterizeImage, waiting on task 912 parsl.dataflow.dflow DEBUG: Task 913 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 913 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 914 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 914 submitted for App calibrate, waiting on task 913 parsl.dataflow.dflow DEBUG: Task 914 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 914 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 915 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 915 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 915 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 915 has memoization hash f6ea43aa2b4dd7159693ba7a33d7251a parsl.dataflow.memoization INFO: Task 915 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4549a480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 915 try 0 launched on executor multi with executor id 306 parsl.dataflow.dflow INFO: Standard output for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/a9aadfea-2213-4059-bc4b-ce9054709382_isr_855136_57.stdout parsl.dataflow.dflow INFO: Standard error for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/a9aadfea-2213-4059-bc4b-ce9054709382_isr_855136_57.stderr parsl.dataflow.dflow DEBUG: Task 916 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 916 submitted for App characterizeImage, waiting on task 915 parsl.dataflow.dflow DEBUG: Task 916 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 916 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 917 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 917 submitted for App calibrate, waiting on task 916 parsl.dataflow.dflow DEBUG: Task 917 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 917 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 918 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 918 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 918 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 918 has memoization hash 39b3061e6aec7d4d6a08d84b158bf907 parsl.dataflow.memoization INFO: Task 918 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4549a7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 918 try 0 launched on executor multi with executor id 307 parsl.dataflow.dflow INFO: Standard output for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/fd7e5c63-97d5-4b6b-942b-194d478822e1_isr_855044_2.stdout parsl.dataflow.dflow INFO: Standard error for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/fd7e5c63-97d5-4b6b-942b-194d478822e1_isr_855044_2.stderr parsl.dataflow.dflow DEBUG: Task 919 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 919 submitted for App characterizeImage, waiting on task 918 parsl.dataflow.dflow DEBUG: Task 919 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 919 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 920 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 920 submitted for App calibrate, waiting on task 919 parsl.dataflow.dflow DEBUG: Task 920 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 920 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 921 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 921 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 921 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 921 has memoization hash 40946beb15cbd1574d2bc0d3f90d8226 parsl.dataflow.memoization INFO: Task 921 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4549aac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 921 try 0 launched on executor multi with executor id 308 parsl.dataflow.dflow INFO: Standard output for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/6d3fd601-7bd5-4aab-8683-62ba84837909_isr_855044_55.stdout parsl.dataflow.dflow INFO: Standard error for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/6d3fd601-7bd5-4aab-8683-62ba84837909_isr_855044_55.stderr parsl.dataflow.dflow DEBUG: Task 922 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 922 submitted for App characterizeImage, waiting on task 921 parsl.dataflow.dflow DEBUG: Task 922 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 922 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 923 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 923 submitted for App calibrate, waiting on task 922 parsl.dataflow.dflow DEBUG: Task 923 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 923 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 924 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 924 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 924 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 924 has memoization hash 83ab448b79951239800ddddddc4ce222 parsl.dataflow.memoization INFO: Task 924 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4549ade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-08 07:15:14 proc_lsst.multi:170 [INFO] [multi] job_id 16912289 proc_lsst.multi INFO: [multi] job_id 16912289 2024-03-08 07:15:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-08 07:15:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:14 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16912289 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16912289 parsl.executors.status_handling DEBUG: Launched block 7 on executor multi with job ID 16912289 parsl.executors.status_handling INFO: Allocated block ID 8 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-8 2024-03-08 07:15:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 2024-03-08 07:15:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-08 07:15:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 924 try 0 launched on executor multi with executor id 309 parsl.dataflow.dflow INFO: Standard output for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/5c713426-550a-4a5f-9b8e-bd9d62fdab7d_isr_855138_46.stdout parsl.dataflow.dflow INFO: Standard error for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/5c713426-550a-4a5f-9b8e-bd9d62fdab7d_isr_855138_46.stderr parsl.dataflow.dflow DEBUG: Task 925 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 925 submitted for App characterizeImage, waiting on task 924 parsl.dataflow.dflow DEBUG: Task 925 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 925 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 926 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 926 submitted for App calibrate, waiting on task 925 parsl.dataflow.dflow DEBUG: Task 926 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 926 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 927 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 927 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 927 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 927 has memoization hash 9cd237fed667841dd3b2340878acc972 parsl.dataflow.memoization INFO: Task 927 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4549ae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 927 try 0 launched on executor multi with executor id 310 parsl.dataflow.dflow INFO: Standard output for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/1116ab67-b71d-4ef4-ade9-e59c6e69beee_isr_855138_25.stdout parsl.dataflow.dflow INFO: Standard error for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/1116ab67-b71d-4ef4-ade9-e59c6e69beee_isr_855138_25.stderr parsl.dataflow.dflow DEBUG: Task 928 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 928 submitted for App characterizeImage, waiting on task 927 parsl.dataflow.dflow DEBUG: Task 928 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 928 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 929 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 929 submitted for App calibrate, waiting on task 928 parsl.dataflow.dflow DEBUG: Task 929 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 929 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 930 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 930 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 930 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 930 has memoization hash 4972e3e94839bdcc6cfddfc783612622 parsl.dataflow.memoization INFO: Task 930 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4549b240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 930 try 0 launched on executor multi with executor id 311 parsl.dataflow.dflow INFO: Standard output for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/47f067b7-1c03-4e71-ae3e-caefbf9f0305_isr_855136_55.stdout parsl.dataflow.dflow INFO: Standard error for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/47f067b7-1c03-4e71-ae3e-caefbf9f0305_isr_855136_55.stderr parsl.dataflow.dflow DEBUG: Task 931 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 931 submitted for App characterizeImage, waiting on task 930 parsl.dataflow.dflow DEBUG: Task 931 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 931 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 932 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 932 submitted for App calibrate, waiting on task 931 parsl.dataflow.dflow DEBUG: Task 932 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 932 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 933 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 933 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 933 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 933 has memoization hash 645a2ea5a3385df82d178f943c2852f5 parsl.dataflow.memoization INFO: Task 933 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4549b560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 933 try 0 launched on executor multi with executor id 312 parsl.dataflow.dflow INFO: Standard output for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/829d50d6-3168-4350-a68d-e2babdf13c00_isr_855137_54.stdout parsl.dataflow.dflow INFO: Standard error for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/829d50d6-3168-4350-a68d-e2babdf13c00_isr_855137_54.stderr parsl.dataflow.dflow DEBUG: Task 934 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 934 submitted for App characterizeImage, waiting on task 933 parsl.dataflow.dflow DEBUG: Task 934 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 934 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 935 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 935 submitted for App calibrate, waiting on task 934 parsl.dataflow.dflow DEBUG: Task 935 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 935 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 936 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 936 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 936 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 936 has memoization hash 927c2a7276f1e6bdcf0965cc3b4d12dd parsl.dataflow.memoization INFO: Task 936 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4549b880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 936 try 0 launched on executor multi with executor id 313 parsl.dataflow.dflow INFO: Standard output for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/3496b1f0-35af-487c-9a9b-d5154807d2ba_isr_855138_17.stdout parsl.dataflow.dflow INFO: Standard error for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/3496b1f0-35af-487c-9a9b-d5154807d2ba_isr_855138_17.stderr parsl.dataflow.dflow DEBUG: Task 937 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 937 submitted for App characterizeImage, waiting on task 936 parsl.dataflow.dflow DEBUG: Task 937 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 937 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 938 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 938 submitted for App calibrate, waiting on task 937 parsl.dataflow.dflow DEBUG: Task 938 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 938 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 939 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 939 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 939 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 939 has memoization hash a906d609c4de5880569be70617bf39a0 parsl.dataflow.memoization INFO: Task 939 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4549bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 939 try 0 launched on executor multi with executor id 314 parsl.dataflow.dflow INFO: Standard output for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/9d502c2e-a165-4311-abd0-1e86df1685f8_isr_855138_30.stdout parsl.dataflow.dflow INFO: Standard error for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/9d502c2e-a165-4311-abd0-1e86df1685f8_isr_855138_30.stderr parsl.dataflow.dflow DEBUG: Task 940 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 940 submitted for App characterizeImage, waiting on task 939 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 940 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 940 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 941 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 941 submitted for App calibrate, waiting on task 940 parsl.dataflow.dflow DEBUG: Task 941 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 941 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 942 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 942 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 942 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 942 has memoization hash d577ba2275719421ad79269f3640274d parsl.dataflow.memoization INFO: Task 942 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454faf20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 942 try 0 launched on executor multi with executor id 315 parsl.dataflow.dflow INFO: Standard output for task 942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/3b460f95-20df-4725-84fe-f427d9674572_isr_855138_22.stdout parsl.dataflow.dflow INFO: Standard error for task 942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/3b460f95-20df-4725-84fe-f427d9674572_isr_855138_22.stderr parsl.dataflow.dflow DEBUG: Task 943 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 943 submitted for App characterizeImage, waiting on task 942 parsl.dataflow.dflow DEBUG: Task 943 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 943 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 944 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 944 submitted for App calibrate, waiting on task 943 parsl.dataflow.dflow DEBUG: Task 944 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 944 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 945 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 945 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 945 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 945 has memoization hash 4013b9a464b599cca22b3bf190be95ba parsl.dataflow.memoization INFO: Task 945 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454fac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 945 try 0 launched on executor multi with executor id 316 parsl.dataflow.dflow INFO: Standard output for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/eed15330-fa29-486a-8334-e80ec6c94326_isr_855136_22.stdout parsl.dataflow.dflow INFO: Standard error for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/eed15330-fa29-486a-8334-e80ec6c94326_isr_855136_22.stderr parsl.dataflow.dflow DEBUG: Task 946 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 946 submitted for App characterizeImage, waiting on task 945 parsl.dataflow.dflow DEBUG: Task 946 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 946 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 947 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 947 submitted for App calibrate, waiting on task 946 parsl.dataflow.dflow DEBUG: Task 947 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 947 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 948 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 948 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 948 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 948 has memoization hash 9b618131f3228a0071191b97bc78d77f parsl.dataflow.memoization INFO: Task 948 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454fa8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 948 try 0 launched on executor multi with executor id 317 parsl.dataflow.dflow INFO: Standard output for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/584fa0d8-1059-4183-99a3-ebf4765a5285_isr_855136_58.stdout parsl.dataflow.dflow INFO: Standard error for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/584fa0d8-1059-4183-99a3-ebf4765a5285_isr_855136_58.stderr parsl.dataflow.dflow DEBUG: Task 949 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 949 submitted for App characterizeImage, waiting on task 948 parsl.dataflow.dflow DEBUG: Task 949 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 949 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 950 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 950 submitted for App calibrate, waiting on task 949 parsl.dataflow.dflow DEBUG: Task 950 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 950 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 951 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 951 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 951 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 951 has memoization hash ab2b4893a643b0bb51f278a0e3fa6e89 parsl.dataflow.memoization INFO: Task 951 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454fa5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 951 try 0 launched on executor multi with executor id 318 parsl.dataflow.dflow INFO: Standard output for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/6eae4fe6-04c6-49c7-95cd-4827cd27f062_isr_855042_11.stdout parsl.dataflow.dflow INFO: Standard error for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/6eae4fe6-04c6-49c7-95cd-4827cd27f062_isr_855042_11.stderr parsl.dataflow.dflow DEBUG: Task 952 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 952 submitted for App characterizeImage, waiting on task 951 parsl.dataflow.dflow DEBUG: Task 952 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 952 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 953 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 953 submitted for App calibrate, waiting on task 952 parsl.dataflow.dflow DEBUG: Task 953 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 953 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 954 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 954 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 954 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 954 has memoization hash 4661f4f20defc6fea74643189fb9b58d parsl.dataflow.memoization INFO: Task 954 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454fa2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 954 try 0 launched on executor multi with executor id 319 parsl.dataflow.dflow INFO: Standard output for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/a1e9cd58-8335-4862-ad93-47a0d0e69386_isr_855138_7.stdout parsl.dataflow.dflow INFO: Standard error for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/a1e9cd58-8335-4862-ad93-47a0d0e69386_isr_855138_7.stderr parsl.dataflow.dflow DEBUG: Task 955 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 955 submitted for App characterizeImage, waiting on task 954 parsl.dataflow.dflow DEBUG: Task 955 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 955 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 956 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 956 submitted for App calibrate, waiting on task 955 parsl.dataflow.dflow DEBUG: Task 956 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 956 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 957 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 957 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 957 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 957 has memoization hash 98c7eb63b1c5356a33da96e1012ed4d8 parsl.dataflow.memoization INFO: Task 957 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454fa200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 957 try 0 launched on executor multi with executor id 320 parsl.dataflow.dflow INFO: Standard output for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/f9835ac6-0f43-47c8-b83e-f9fabb593841_isr_855137_22.stdout parsl.dataflow.dflow INFO: Standard error for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/f9835ac6-0f43-47c8-b83e-f9fabb593841_isr_855137_22.stderr parsl.dataflow.dflow DEBUG: Task 958 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 958 submitted for App characterizeImage, waiting on task 957 parsl.dataflow.dflow DEBUG: Task 958 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 958 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 959 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 959 submitted for App calibrate, waiting on task 958 parsl.dataflow.dflow DEBUG: Task 959 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 959 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 960 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 960 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 960 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 960 has memoization hash a649b524262c9138b618eb64f598ee5b parsl.dataflow.memoization INFO: Task 960 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454f8220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 960 try 0 launched on executor multi with executor id 321 parsl.dataflow.dflow INFO: Standard output for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/542c9f54-dcc9-4d14-a31c-c5a43324fe0e_isr_855043_27.stdout parsl.dataflow.dflow INFO: Standard error for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/542c9f54-dcc9-4d14-a31c-c5a43324fe0e_isr_855043_27.stderr parsl.dataflow.dflow DEBUG: Task 961 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 961 submitted for App characterizeImage, waiting on task 960 parsl.dataflow.dflow DEBUG: Task 961 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 961 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 962 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 962 submitted for App calibrate, waiting on task 961 parsl.dataflow.dflow DEBUG: Task 962 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 962 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 963 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 963 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 963 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 963 has memoization hash 2cbf920d1b4eb6d769bfbbc92f0230ee parsl.dataflow.memoization INFO: Task 963 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454f8400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 963 try 0 launched on executor multi with executor id 322 parsl.dataflow.dflow INFO: Standard output for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/55c3a70a-4f39-446c-b1da-66ad4abf9b63_isr_855043_54.stdout parsl.dataflow.dflow INFO: Standard error for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/55c3a70a-4f39-446c-b1da-66ad4abf9b63_isr_855043_54.stderr parsl.dataflow.dflow DEBUG: Task 964 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 964 submitted for App characterizeImage, waiting on task 963 parsl.dataflow.dflow DEBUG: Task 964 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 964 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 965 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 965 submitted for App calibrate, waiting on task 964 parsl.dataflow.dflow DEBUG: Task 965 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 965 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 966 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 966 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 966 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 966 has memoization hash e1185578ebcfe468af0816bda25e8e17 parsl.dataflow.memoization INFO: Task 966 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454f8900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 966 try 0 launched on executor multi with executor id 323 parsl.dataflow.dflow INFO: Standard output for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/1547185b-6c45-4697-80a3-14921a62b21c_isr_855043_7.stdout parsl.dataflow.dflow INFO: Standard error for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/1547185b-6c45-4697-80a3-14921a62b21c_isr_855043_7.stderr parsl.dataflow.dflow DEBUG: Task 967 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 967 submitted for App characterizeImage, waiting on task 966 parsl.dataflow.dflow DEBUG: Task 967 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 967 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 968 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 968 submitted for App calibrate, waiting on task 967 parsl.dataflow.dflow DEBUG: Task 968 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 968 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 969 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 969 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 969 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 969 has memoization hash ed053a8fd9124ccf2f6780eb505bd3ce parsl.dataflow.memoization INFO: Task 969 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454f8c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 969 try 0 launched on executor multi with executor id 324 parsl.dataflow.dflow INFO: Standard output for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/2fb5a2b3-9e28-4be0-84dd-ca05ec516fd1_isr_855042_61.stdout parsl.dataflow.dflow INFO: Standard error for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/2fb5a2b3-9e28-4be0-84dd-ca05ec516fd1_isr_855042_61.stderr parsl.dataflow.dflow DEBUG: Task 970 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 970 submitted for App characterizeImage, waiting on task 969 parsl.dataflow.dflow DEBUG: Task 970 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 970 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 971 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 971 submitted for App calibrate, waiting on task 970 parsl.dataflow.dflow DEBUG: Task 971 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 971 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 972 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 972 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 972 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 972 has memoization hash e758e8904626e95a447c5750b76f20ab parsl.dataflow.memoization INFO: Task 972 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454f8ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 972 try 0 launched on executor multi with executor id 325 parsl.dataflow.dflow INFO: Standard output for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/ed34e911-41df-4374-a9a9-3477de67630c_isr_855137_10.stdout parsl.dataflow.dflow INFO: Standard error for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/ed34e911-41df-4374-a9a9-3477de67630c_isr_855137_10.stderr parsl.dataflow.dflow DEBUG: Task 973 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 973 submitted for App characterizeImage, waiting on task 972 parsl.dataflow.dflow DEBUG: Task 973 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 973 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 974 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 974 submitted for App calibrate, waiting on task 973 parsl.dataflow.dflow DEBUG: Task 974 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 974 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 975 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 975 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 975 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 975 has memoization hash a9a8effc13c5988725522d3011f3d4e4 parsl.dataflow.memoization INFO: Task 975 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454f91c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 975 try 0 launched on executor multi with executor id 326 parsl.dataflow.dflow INFO: Standard output for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/c4ef022f-3184-4fe9-9888-ce3994969f79_isr_855044_59.stdout parsl.dataflow.dflow INFO: Standard error for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/c4ef022f-3184-4fe9-9888-ce3994969f79_isr_855044_59.stderr parsl.dataflow.dflow DEBUG: Task 976 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 976 submitted for App characterizeImage, waiting on task 975 parsl.dataflow.dflow DEBUG: Task 976 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 976 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 977 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 977 submitted for App calibrate, waiting on task 976 parsl.dataflow.dflow DEBUG: Task 977 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 977 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 978 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 978 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 978 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 978 has memoization hash b08b0b20ef07311bec6f792b9cc9e37c parsl.dataflow.memoization INFO: Task 978 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454f9580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 978 try 0 launched on executor multi with executor id 327 parsl.dataflow.dflow INFO: Standard output for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/0fa03805-a20e-4086-bdda-e9cef76c9ac8_isr_855138_11.stdout parsl.dataflow.dflow INFO: Standard error for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/0fa03805-a20e-4086-bdda-e9cef76c9ac8_isr_855138_11.stderr parsl.dataflow.dflow DEBUG: Task 979 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 979 submitted for App characterizeImage, waiting on task 978 parsl.dataflow.dflow DEBUG: Task 979 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 979 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 980 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 980 submitted for App calibrate, waiting on task 979 parsl.dataflow.dflow DEBUG: Task 980 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 980 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 981 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 981 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 981 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 981 has memoization hash 5db215b4e9a95aa6fec0d01324a1b81a parsl.dataflow.memoization INFO: Task 981 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454f98a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 981 try 0 launched on executor multi with executor id 328 parsl.dataflow.dflow INFO: Standard output for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/f711336e-1e86-456f-8049-0bb08264fb48_isr_855138_31.stdout parsl.dataflow.dflow INFO: Standard error for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/f711336e-1e86-456f-8049-0bb08264fb48_isr_855138_31.stderr parsl.dataflow.dflow DEBUG: Task 982 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 982 submitted for App characterizeImage, waiting on task 981 parsl.dataflow.dflow DEBUG: Task 982 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 982 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 983 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 983 submitted for App calibrate, waiting on task 982 parsl.dataflow.dflow DEBUG: Task 983 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 983 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 984 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 984 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 984 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 984 has memoization hash 2ed533cda13dde14e8b45a8c6460c26d parsl.dataflow.memoization INFO: Task 984 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454f9940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 984 try 0 launched on executor multi with executor id 329 parsl.dataflow.dflow INFO: Standard output for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/08721a85-9aab-4bb9-9ddf-b9b2b5ad213a_isr_855138_20.stdout parsl.dataflow.dflow INFO: Standard error for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/08721a85-9aab-4bb9-9ddf-b9b2b5ad213a_isr_855138_20.stderr parsl.dataflow.dflow DEBUG: Task 985 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 985 submitted for App characterizeImage, waiting on task 984 parsl.dataflow.dflow DEBUG: Task 985 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 985 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 986 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 986 submitted for App calibrate, waiting on task 985 parsl.dataflow.dflow DEBUG: Task 986 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 986 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 987 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 987 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 987 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 987 has memoization hash 5ddcbdef52c365e6b1635502d4aa1bbf parsl.dataflow.memoization INFO: Task 987 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454f9f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 987 try 0 launched on executor multi with executor id 330 parsl.dataflow.dflow INFO: Standard output for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/e569cab5-ef45-4494-b5c6-d272b9adca35_isr_855136_51.stdout parsl.dataflow.dflow INFO: Standard error for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/e569cab5-ef45-4494-b5c6-d272b9adca35_isr_855136_51.stderr parsl.dataflow.dflow DEBUG: Task 988 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 988 submitted for App characterizeImage, waiting on task 987 parsl.dataflow.dflow DEBUG: Task 988 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 988 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 989 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 989 submitted for App calibrate, waiting on task 988 parsl.dataflow.dflow DEBUG: Task 989 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 989 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 990 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 990 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 990 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 990 has memoization hash b065f6c4e7eddb8155678a36ee08f15e parsl.dataflow.memoization INFO: Task 990 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454fb2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 990 try 0 launched on executor multi with executor id 331 parsl.dataflow.dflow INFO: Standard output for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/2972c186-87ef-4341-90b8-7f3a60c72d01_isr_855044_11.stdout parsl.dataflow.dflow INFO: Standard error for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/2972c186-87ef-4341-90b8-7f3a60c72d01_isr_855044_11.stderr parsl.dataflow.dflow DEBUG: Task 991 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 991 submitted for App characterizeImage, waiting on task 990 parsl.dataflow.dflow DEBUG: Task 991 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 991 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 992 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 992 submitted for App calibrate, waiting on task 991 parsl.dataflow.dflow DEBUG: Task 992 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 992 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 993 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 993 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 993 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 993 has memoization hash 0155eb076a73c0b86dc3ea68c26e57bf parsl.dataflow.memoization INFO: Task 993 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454fb600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 993 try 0 launched on executor multi with executor id 332 parsl.dataflow.dflow INFO: Standard output for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/226ab265-e240-485a-b95e-0d96c4b56442_isr_855136_10.stdout parsl.dataflow.dflow INFO: Standard error for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/226ab265-e240-485a-b95e-0d96c4b56442_isr_855136_10.stderr parsl.dataflow.dflow DEBUG: Task 994 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 994 submitted for App characterizeImage, waiting on task 993 parsl.dataflow.dflow DEBUG: Task 994 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 994 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 995 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 995 submitted for App calibrate, waiting on task 994 parsl.dataflow.dflow DEBUG: Task 995 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 995 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 996 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 996 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 996 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 996 has memoization hash 1c01da95e4186b7f002c404625526ce9 parsl.dataflow.memoization INFO: Task 996 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454fb920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 996 try 0 launched on executor multi with executor id 333 parsl.dataflow.dflow INFO: Standard output for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/3606ee25-c31a-4769-9ec3-f8564ab413c8_isr_855138_26.stdout parsl.dataflow.dflow INFO: Standard error for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/3606ee25-c31a-4769-9ec3-f8564ab413c8_isr_855138_26.stderr parsl.dataflow.dflow DEBUG: Task 997 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 997 submitted for App characterizeImage, waiting on task 996 parsl.dataflow.dflow DEBUG: Task 997 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 997 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 998 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 998 submitted for App calibrate, waiting on task 997 parsl.dataflow.dflow DEBUG: Task 998 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 998 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 999 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 999 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 999 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 999 has memoization hash 10983e10acc270eeb45b42d413b816bb parsl.dataflow.memoization INFO: Task 999 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454fbc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 999 try 0 launched on executor multi with executor id 334 parsl.dataflow.dflow INFO: Standard output for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/1b62f5ff-3adf-4c47-a705-f700666d7ce2_isr_855044_49.stdout parsl.dataflow.dflow INFO: Standard error for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/1b62f5ff-3adf-4c47-a705-f700666d7ce2_isr_855044_49.stderr parsl.dataflow.dflow DEBUG: Task 1000 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1000 submitted for App characterizeImage, waiting on task 999 parsl.dataflow.dflow DEBUG: Task 1000 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1000 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1001 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1001 submitted for App calibrate, waiting on task 1000 parsl.dataflow.dflow DEBUG: Task 1001 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1001 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1002 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1002 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1002 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1002 has memoization hash 7a7cb5e3ec1391f043124ce0655d85b9 parsl.dataflow.memoization INFO: Task 1002 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454fbf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1002 try 0 launched on executor multi with executor id 335 parsl.dataflow.dflow INFO: Standard output for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/21a52822-db1d-4fe5-97eb-d946eb5d9ffb_isr_855136_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/21a52822-db1d-4fe5-97eb-d946eb5d9ffb_isr_855136_53.stderr parsl.dataflow.dflow DEBUG: Task 1003 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1003 submitted for App characterizeImage, waiting on task 1002 parsl.dataflow.dflow DEBUG: Task 1003 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1003 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1004 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1004 submitted for App calibrate, waiting on task 1003 parsl.dataflow.dflow DEBUG: Task 1004 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1004 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1005 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1005 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1005 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1005 has memoization hash 7008df90a3226ec351bf10835807ba25 parsl.dataflow.memoization INFO: Task 1005 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455402c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-08 07:15:14 proc_lsst.multi:170 [INFO] [multi] job_id 16912290 proc_lsst.multi INFO: [multi] job_id 16912290 2024-03-08 07:15:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-08 07:15:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:14 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16912290 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16912290 parsl.executors.status_handling DEBUG: Launched block 8 on executor multi with job ID 16912290 parsl.executors.status_handling INFO: Allocated block ID 9 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-9 2024-03-08 07:15:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 2024-03-08 07:15:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-08 07:15:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1005 try 0 launched on executor multi with executor id 336 parsl.dataflow.dflow INFO: Standard output for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/c6bd814d-cbd5-4236-858a-f5a37cd9fd5a_isr_855042_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/c6bd814d-cbd5-4236-858a-f5a37cd9fd5a_isr_855042_23.stderr parsl.dataflow.dflow DEBUG: Task 1006 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1006 submitted for App characterizeImage, waiting on task 1005 parsl.dataflow.dflow DEBUG: Task 1006 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1006 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1007 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1007 submitted for App calibrate, waiting on task 1006 parsl.dataflow.dflow DEBUG: Task 1007 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1007 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1008 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1008 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1008 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1008 has memoization hash 9ab4c2435065fe37ffde973aa7d471ad parsl.dataflow.memoization INFO: Task 1008 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45540360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1008 try 0 launched on executor multi with executor id 337 parsl.dataflow.dflow INFO: Standard output for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/92cbee0e-8d38-420c-b7d6-784a5f5d6ec6_isr_855043_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/92cbee0e-8d38-420c-b7d6-784a5f5d6ec6_isr_855043_24.stderr parsl.dataflow.dflow DEBUG: Task 1009 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1009 submitted for App characterizeImage, waiting on task 1008 parsl.dataflow.dflow DEBUG: Task 1009 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1009 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1010 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1010 submitted for App calibrate, waiting on task 1009 parsl.dataflow.dflow DEBUG: Task 1010 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1010 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1011 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1011 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1011 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1011 has memoization hash 74f5128366cffe96b15e166e9feb7955 parsl.dataflow.memoization INFO: Task 1011 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455439c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1011 try 0 launched on executor multi with executor id 338 parsl.dataflow.dflow INFO: Standard output for task 1011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/fe4c95c4-a670-4184-8ac4-52fa10c6a2ee_isr_855138_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/fe4c95c4-a670-4184-8ac4-52fa10c6a2ee_isr_855138_59.stderr parsl.dataflow.dflow DEBUG: Task 1012 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1012 submitted for App characterizeImage, waiting on task 1011 parsl.dataflow.dflow DEBUG: Task 1012 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1012 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1013 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1013 submitted for App calibrate, waiting on task 1012 parsl.dataflow.dflow DEBUG: Task 1013 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1013 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1014 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1014 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1014 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1014 has memoization hash 3e3ad07164a07c4dbea96ead8cd398d0 parsl.dataflow.memoization INFO: Task 1014 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45540860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1014 try 0 launched on executor multi with executor id 339 parsl.dataflow.dflow INFO: Standard output for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/c0bb59ce-6e0d-4f4a-83a0-09d65537bccc_isr_855138_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/c0bb59ce-6e0d-4f4a-83a0-09d65537bccc_isr_855138_52.stderr parsl.dataflow.dflow DEBUG: Task 1015 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1015 submitted for App characterizeImage, waiting on task 1014 parsl.dataflow.dflow DEBUG: Task 1015 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1015 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1016 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1016 submitted for App calibrate, waiting on task 1015 parsl.dataflow.dflow DEBUG: Task 1016 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1016 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1017 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1017 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1017 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1017 has memoization hash b4e536a041c9ec4c4b4ec338d31e3267 parsl.dataflow.memoization INFO: Task 1017 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45540c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1017 try 0 launched on executor multi with executor id 340 parsl.dataflow.dflow INFO: Standard output for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/79c4b19a-a445-4378-8983-aab03df78370_isr_855137_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/79c4b19a-a445-4378-8983-aab03df78370_isr_855137_24.stderr parsl.dataflow.dflow DEBUG: Task 1018 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1018 submitted for App characterizeImage, waiting on task 1017 parsl.dataflow.dflow DEBUG: Task 1018 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1018 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1019 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1019 submitted for App calibrate, waiting on task 1018 parsl.dataflow.dflow DEBUG: Task 1019 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1019 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1020 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1020 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1020 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1020 has memoization hash 0649aefc3472bb6e31d36fa43f825bd3 parsl.dataflow.memoization INFO: Task 1020 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45540e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1020 try 0 launched on executor multi with executor id 341 parsl.dataflow.dflow INFO: Standard output for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/28f2e87d-415d-46c1-b263-c9cad74eb93f_isr_855042_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/28f2e87d-415d-46c1-b263-c9cad74eb93f_isr_855042_60.stderr parsl.dataflow.dflow DEBUG: Task 1021 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1021 submitted for App characterizeImage, waiting on task 1020 parsl.dataflow.dflow DEBUG: Task 1021 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1021 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1022 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1022 submitted for App calibrate, waiting on task 1021 parsl.dataflow.dflow DEBUG: Task 1022 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1022 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1023 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1023 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1023 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1023 has memoization hash 7c6e01a79ef47cea1257b6552bbafe7f parsl.dataflow.memoization INFO: Task 1023 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45541300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1023 try 0 launched on executor multi with executor id 342 parsl.dataflow.dflow INFO: Standard output for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/936fe576-4d39-4579-b7cd-e9379d29836a_isr_855043_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/936fe576-4d39-4579-b7cd-e9379d29836a_isr_855043_11.stderr parsl.dataflow.dflow DEBUG: Task 1024 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1024 submitted for App characterizeImage, waiting on task 1023 parsl.dataflow.dflow DEBUG: Task 1024 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1024 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1025 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1025 submitted for App calibrate, waiting on task 1024 parsl.dataflow.dflow DEBUG: Task 1025 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1025 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1026 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1026 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1026 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1026 has memoization hash 2dc81e1e980550f8abdecfcf44b45c42 parsl.dataflow.memoization INFO: Task 1026 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45541620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1026 try 0 launched on executor multi with executor id 343 parsl.dataflow.dflow INFO: Standard output for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/fadbdff8-8e84-4a26-b7e3-7045401ee5e8_isr_855042_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/fadbdff8-8e84-4a26-b7e3-7045401ee5e8_isr_855042_56.stderr parsl.dataflow.dflow DEBUG: Task 1027 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1027 submitted for App characterizeImage, waiting on task 1026 parsl.dataflow.dflow DEBUG: Task 1027 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1027 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1028 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1028 submitted for App calibrate, waiting on task 1027 parsl.dataflow.dflow DEBUG: Task 1028 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1028 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1029 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1029 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1029 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1029 has memoization hash 924c3071a231e25bb03ccb71075ceabd parsl.dataflow.memoization INFO: Task 1029 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455418a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1029 try 0 launched on executor multi with executor id 344 parsl.dataflow.dflow INFO: Standard output for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/1fd4a95f-e0a5-47e2-8b72-c74b08ac9845_isr_855042_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/1fd4a95f-e0a5-47e2-8b72-c74b08ac9845_isr_855042_40.stderr parsl.dataflow.dflow DEBUG: Task 1030 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1030 submitted for App characterizeImage, waiting on task 1029 parsl.dataflow.dflow DEBUG: Task 1030 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1030 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1031 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1031 submitted for App calibrate, waiting on task 1030 parsl.dataflow.dflow DEBUG: Task 1031 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1031 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1032 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1032 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1032 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1032 has memoization hash cb252179bc75a74ed359abb0c6860f9d parsl.dataflow.memoization INFO: Task 1032 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45541bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1032 try 0 launched on executor multi with executor id 345 parsl.dataflow.dflow INFO: Standard output for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/0ef5d4d9-bfd3-4dac-8a75-89f82462f9fc_isr_855043_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/0ef5d4d9-bfd3-4dac-8a75-89f82462f9fc_isr_855043_38.stderr parsl.dataflow.dflow DEBUG: Task 1033 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1033 submitted for App characterizeImage, waiting on task 1032 parsl.dataflow.dflow DEBUG: Task 1033 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1033 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1034 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1034 submitted for App calibrate, waiting on task 1033 parsl.dataflow.dflow DEBUG: Task 1034 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1034 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1035 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1035 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1035 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1035 has memoization hash 622b2b62eaf673638731604ebb69e3c7 parsl.dataflow.memoization INFO: Task 1035 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45541f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1035 try 0 launched on executor multi with executor id 346 parsl.dataflow.dflow INFO: Standard output for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/c7db2321-9f9d-446d-b129-3e22770173c6_isr_855136_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/c7db2321-9f9d-446d-b129-3e22770173c6_isr_855136_54.stderr parsl.dataflow.dflow DEBUG: Task 1036 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1036 submitted for App characterizeImage, waiting on task 1035 parsl.dataflow.dflow DEBUG: Task 1036 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1036 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1037 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1037 submitted for App calibrate, waiting on task 1036 parsl.dataflow.dflow DEBUG: Task 1037 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1037 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1038 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1038 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1038 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1038 has memoization hash 5190eea96c4199376e0da29d9d4d2f00 parsl.dataflow.memoization INFO: Task 1038 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455422a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1038 try 0 launched on executor multi with executor id 347 parsl.dataflow.dflow INFO: Standard output for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/9eb3c489-c634-4089-ac86-dd19d00e6af2_isr_855138_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/9eb3c489-c634-4089-ac86-dd19d00e6af2_isr_855138_61.stderr parsl.dataflow.dflow DEBUG: Task 1039 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1039 submitted for App characterizeImage, waiting on task 1038 parsl.dataflow.dflow DEBUG: Task 1039 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1039 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1040 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1040 submitted for App calibrate, waiting on task 1039 parsl.dataflow.dflow DEBUG: Task 1040 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1040 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1041 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1041 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1041 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1041 has memoization hash 63550a08f0af41f3cf34c9fb19b52bae parsl.dataflow.memoization INFO: Task 1041 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45542340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1041 try 0 launched on executor multi with executor id 348 parsl.dataflow.dflow INFO: Standard output for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/15526cc5-18e7-47cb-bc75-3733d361b914_isr_855136_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/15526cc5-18e7-47cb-bc75-3733d361b914_isr_855136_16.stderr parsl.dataflow.dflow DEBUG: Task 1042 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1042 submitted for App characterizeImage, waiting on task 1041 parsl.dataflow.dflow DEBUG: Task 1042 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1042 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1043 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1043 submitted for App calibrate, waiting on task 1042 parsl.dataflow.dflow DEBUG: Task 1043 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1043 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1044 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1044 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1044 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1044 has memoization hash 7602e7b5b413f7e1e6da832650f7c023 parsl.dataflow.memoization INFO: Task 1044 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455436a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1044 try 0 launched on executor multi with executor id 349 parsl.dataflow.dflow INFO: Standard output for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/09f2bc0c-f482-48ed-b70e-d3837454400e_isr_855044_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/09f2bc0c-f482-48ed-b70e-d3837454400e_isr_855044_54.stderr parsl.dataflow.dflow DEBUG: Task 1045 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1045 submitted for App characterizeImage, waiting on task 1044 parsl.dataflow.dflow DEBUG: Task 1045 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1045 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1046 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1046 submitted for App calibrate, waiting on task 1045 parsl.dataflow.dflow DEBUG: Task 1046 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1046 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1047 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1047 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1047 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1047 has memoization hash 40949c8d69ba4eed70ae69167c699778 parsl.dataflow.memoization INFO: Task 1047 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45543380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1047 try 0 launched on executor multi with executor id 350 parsl.dataflow.dflow INFO: Standard output for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/4107f3d6-babe-4a94-8752-49db50fb6ace_isr_855137_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/4107f3d6-babe-4a94-8752-49db50fb6ace_isr_855137_9.stderr parsl.dataflow.dflow DEBUG: Task 1048 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1048 submitted for App characterizeImage, waiting on task 1047 parsl.dataflow.dflow DEBUG: Task 1048 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1048 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1049 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1049 submitted for App calibrate, waiting on task 1048 parsl.dataflow.dflow DEBUG: Task 1049 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1049 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1050 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1050 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1050 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1050 has memoization hash 91ffa259569aa3d551508f99528187e0 parsl.dataflow.memoization INFO: Task 1050 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45543060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1050 try 0 launched on executor multi with executor id 351 parsl.dataflow.dflow INFO: Standard output for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/ae8f296e-6e5c-4a25-bf12-f511859f6e2e_isr_855043_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/ae8f296e-6e5c-4a25-bf12-f511859f6e2e_isr_855043_51.stderr parsl.dataflow.dflow DEBUG: Task 1051 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1051 submitted for App characterizeImage, waiting on task 1050 parsl.dataflow.dflow DEBUG: Task 1051 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1051 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1052 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1052 submitted for App calibrate, waiting on task 1051 parsl.dataflow.dflow DEBUG: Task 1052 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1052 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1053 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1053 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1053 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1053 has memoization hash ae28f44aec13179e12bcb17a86e1b661 parsl.dataflow.memoization INFO: Task 1053 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45542d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1053 try 0 launched on executor multi with executor id 352 parsl.dataflow.dflow INFO: Standard output for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/2236bb40-2430-4463-82e8-66c9c0810c68_isr_855043_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/2236bb40-2430-4463-82e8-66c9c0810c68_isr_855043_9.stderr parsl.dataflow.dflow DEBUG: Task 1054 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1054 submitted for App characterizeImage, waiting on task 1053 parsl.dataflow.dflow DEBUG: Task 1054 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1054 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1055 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1055 submitted for App calibrate, waiting on task 1054 parsl.dataflow.dflow DEBUG: Task 1055 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1055 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1056 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1056 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1056 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1056 has memoization hash a727423a2f55aa49a2aef0d076ec575f parsl.dataflow.memoization INFO: Task 1056 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45542a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1056 try 0 launched on executor multi with executor id 353 parsl.dataflow.dflow INFO: Standard output for task 1056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/d6f52d2c-88a7-4799-b63f-2c52fd39028a_isr_855042_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/d6f52d2c-88a7-4799-b63f-2c52fd39028a_isr_855042_37.stderr parsl.dataflow.dflow DEBUG: Task 1057 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1057 submitted for App characterizeImage, waiting on task 1056 parsl.dataflow.dflow DEBUG: Task 1057 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1057 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1058 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1058 submitted for App calibrate, waiting on task 1057 parsl.dataflow.dflow DEBUG: Task 1058 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1058 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1059 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1059 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1059 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1059 has memoization hash 3e5c1dfab2b86f2d0eb5bf13aba0106c parsl.dataflow.memoization INFO: Task 1059 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455404a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-08 07:15:14 proc_lsst.multi:170 [INFO] [multi] job_id 16912291 proc_lsst.multi INFO: [multi] job_id 16912291 2024-03-08 07:15:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-08 07:15:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:14 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16912291 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16912291 parsl.executors.status_handling DEBUG: Launched block 9 on executor multi with job ID 16912291 parsl.executors.status_handling INFO: Allocated block ID 10 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-10 2024-03-08 07:15:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 2024-03-08 07:15:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-08 07:15:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1059 try 0 launched on executor multi with executor id 354 parsl.dataflow.dflow INFO: Standard output for task 1059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/57991402-97d2-4cd0-bf5e-22f4c97d5903_isr_855137_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/57991402-97d2-4cd0-bf5e-22f4c97d5903_isr_855137_11.stderr parsl.dataflow.dflow DEBUG: Task 1060 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1060 submitted for App characterizeImage, waiting on task 1059 parsl.dataflow.dflow DEBUG: Task 1060 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1060 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1061 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1061 submitted for App calibrate, waiting on task 1060 parsl.dataflow.dflow DEBUG: Task 1061 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1061 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1062 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1062 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1062 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1062 has memoization hash 3d8b68465a3c0d4f23a0d025deee951e parsl.dataflow.memoization INFO: Task 1062 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455407c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1062 try 0 launched on executor multi with executor id 355 parsl.dataflow.dflow INFO: Standard output for task 1062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/3a598482-7f66-4a83-b463-4242aaa61377_isr_855044_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/3a598482-7f66-4a83-b463-4242aaa61377_isr_855044_19.stderr parsl.dataflow.dflow DEBUG: Task 1063 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1063 submitted for App characterizeImage, waiting on task 1062 parsl.dataflow.dflow DEBUG: Task 1063 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1063 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1064 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1064 submitted for App calibrate, waiting on task 1063 parsl.dataflow.dflow DEBUG: Task 1064 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1064 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1065 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1065 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1065 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1065 has memoization hash 67ab182f690618bf3f50b51e3d588c42 parsl.dataflow.memoization INFO: Task 1065 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45393380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1065 try 0 launched on executor multi with executor id 356 parsl.dataflow.dflow INFO: Standard output for task 1065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/4ee468d3-09cf-44b8-9218-d745d75011cc_isr_855138_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/4ee468d3-09cf-44b8-9218-d745d75011cc_isr_855138_19.stderr parsl.dataflow.dflow DEBUG: Task 1066 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1066 submitted for App characterizeImage, waiting on task 1065 parsl.dataflow.dflow DEBUG: Task 1066 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1066 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1067 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1067 submitted for App calibrate, waiting on task 1066 parsl.dataflow.dflow DEBUG: Task 1067 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1067 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1068 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1068 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1068 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1068 has memoization hash 7a1656c4c4ba39497a6ff916f34b95c7 parsl.dataflow.memoization INFO: Task 1068 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45393060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1068 try 0 launched on executor multi with executor id 357 parsl.dataflow.dflow INFO: Standard output for task 1068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/de1ff9c8-a419-4670-a93b-5817763c7be9_isr_855138_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/de1ff9c8-a419-4670-a93b-5817763c7be9_isr_855138_36.stderr parsl.dataflow.dflow DEBUG: Task 1069 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1069 submitted for App characterizeImage, waiting on task 1068 parsl.dataflow.dflow DEBUG: Task 1069 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1069 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1070 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1070 submitted for App calibrate, waiting on task 1069 parsl.dataflow.dflow DEBUG: Task 1070 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1070 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1071 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1071 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1071 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1071 has memoization hash a5f877bd45bc611dade83ac4c4dbddcd parsl.dataflow.memoization INFO: Task 1071 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45392d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1071 try 0 launched on executor multi with executor id 358 parsl.dataflow.dflow INFO: Standard output for task 1071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/e4b007e3-17ac-44be-ab15-10976b0d0977_isr_855136_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/e4b007e3-17ac-44be-ab15-10976b0d0977_isr_855136_24.stderr parsl.dataflow.dflow DEBUG: Task 1072 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1072 submitted for App characterizeImage, waiting on task 1071 parsl.dataflow.dflow DEBUG: Task 1072 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1072 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1073 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1073 submitted for App calibrate, waiting on task 1072 parsl.dataflow.dflow DEBUG: Task 1073 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1073 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1074 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1074 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1074 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1074 has memoization hash 6f9f8a2a4441c6a9e8da9cc152b5bd28 parsl.dataflow.memoization INFO: Task 1074 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45392a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1074 try 0 launched on executor multi with executor id 359 parsl.dataflow.dflow INFO: Standard output for task 1074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/3d67f33f-f37f-49c7-a431-a83ae2dd3158_isr_855136_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/3d67f33f-f37f-49c7-a431-a83ae2dd3158_isr_855136_46.stderr parsl.dataflow.dflow DEBUG: Task 1075 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1075 submitted for App characterizeImage, waiting on task 1074 parsl.dataflow.dflow DEBUG: Task 1075 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1075 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1076 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1076 submitted for App calibrate, waiting on task 1075 parsl.dataflow.dflow DEBUG: Task 1076 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1076 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1077 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1077 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1077 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1077 has memoization hash 9c47e6187610fd143ce9b044cefb8653 parsl.dataflow.memoization INFO: Task 1077 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45392700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1077 try 0 launched on executor multi with executor id 360 parsl.dataflow.dflow INFO: Standard output for task 1077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/91d6f87d-593b-4ae3-804b-4aa795923863_isr_855138_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/91d6f87d-593b-4ae3-804b-4aa795923863_isr_855138_28.stderr parsl.dataflow.dflow DEBUG: Task 1078 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1078 submitted for App characterizeImage, waiting on task 1077 parsl.dataflow.dflow DEBUG: Task 1078 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1078 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1079 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1079 submitted for App calibrate, waiting on task 1078 parsl.dataflow.dflow DEBUG: Task 1079 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1079 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1080 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1080 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1080 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1080 has memoization hash d03be96244256fd6dc38935cd2a8e58e parsl.dataflow.memoization INFO: Task 1080 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45390540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1080 try 0 launched on executor multi with executor id 361 parsl.dataflow.dflow INFO: Standard output for task 1080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/0906a5c5-e998-4970-a860-b9d5ef14a426_isr_855043_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/0906a5c5-e998-4970-a860-b9d5ef14a426_isr_855043_22.stderr parsl.dataflow.dflow DEBUG: Task 1081 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1081 submitted for App characterizeImage, waiting on task 1080 parsl.dataflow.dflow DEBUG: Task 1081 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1081 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1082 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1082 submitted for App calibrate, waiting on task 1081 parsl.dataflow.dflow DEBUG: Task 1082 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1082 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1083 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1083 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1083 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1083 has memoization hash 96c315dc3caf57d12d0ad4c175e6c833 parsl.dataflow.memoization INFO: Task 1083 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45390860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1083 try 0 launched on executor multi with executor id 362 parsl.dataflow.dflow INFO: Standard output for task 1083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/373dcf03-4020-4e77-afec-8c33ce55cbe7_isr_855136_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/373dcf03-4020-4e77-afec-8c33ce55cbe7_isr_855136_19.stderr parsl.dataflow.dflow DEBUG: Task 1084 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1084 submitted for App characterizeImage, waiting on task 1083 parsl.dataflow.dflow DEBUG: Task 1084 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1084 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1085 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1085 submitted for App calibrate, waiting on task 1084 parsl.dataflow.dflow DEBUG: Task 1085 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1085 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1086 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1086 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1086 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1086 has memoization hash 28c051a01cb4108b0e6f91f57122ce56 parsl.dataflow.memoization INFO: Task 1086 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45390c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1086 try 0 launched on executor multi with executor id 363 parsl.dataflow.dflow INFO: Standard output for task 1086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/952abdf3-8f1b-4297-8189-6144b865bad8_isr_855136_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/952abdf3-8f1b-4297-8189-6144b865bad8_isr_855136_56.stderr parsl.dataflow.dflow DEBUG: Task 1087 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1087 submitted for App characterizeImage, waiting on task 1086 parsl.dataflow.dflow DEBUG: Task 1087 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1087 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1088 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1088 submitted for App calibrate, waiting on task 1087 parsl.dataflow.dflow DEBUG: Task 1088 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1088 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1089 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1089 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1089 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1089 has memoization hash 541c95755d58dfec5e805be55203df78 parsl.dataflow.memoization INFO: Task 1089 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45390f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1089 try 0 launched on executor multi with executor id 364 parsl.dataflow.dflow INFO: Standard output for task 1089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/1bd78a6a-276c-4092-95f1-9e4de9e21321_isr_855136_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/1bd78a6a-276c-4092-95f1-9e4de9e21321_isr_855136_17.stderr parsl.dataflow.dflow DEBUG: Task 1090 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1090 submitted for App characterizeImage, waiting on task 1089 parsl.dataflow.dflow DEBUG: Task 1090 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1090 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1091 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1091 submitted for App calibrate, waiting on task 1090 parsl.dataflow.dflow DEBUG: Task 1091 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1091 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1092 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1092 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1092 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1092 has memoization hash a5b732ba1939bd5d580d301824833180 parsl.dataflow.memoization INFO: Task 1092 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45391260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1092 try 0 launched on executor multi with executor id 365 parsl.dataflow.dflow INFO: Standard output for task 1092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/eca75e3c-41c8-4245-a78c-ea0a7663fd59_isr_855043_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/eca75e3c-41c8-4245-a78c-ea0a7663fd59_isr_855043_23.stderr parsl.dataflow.dflow DEBUG: Task 1093 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1093 submitted for App characterizeImage, waiting on task 1092 parsl.dataflow.dflow DEBUG: Task 1093 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1093 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1094 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1094 submitted for App calibrate, waiting on task 1093 parsl.dataflow.dflow DEBUG: Task 1094 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1094 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1095 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1095 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1095 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1095 has memoization hash 6b1a216300c38dfa200a74b411392a4b parsl.dataflow.memoization INFO: Task 1095 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45391580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1095 try 0 launched on executor multi with executor id 366 parsl.dataflow.dflow INFO: Standard output for task 1095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/f51a547a-90c2-42ef-8196-15c2575da80c_isr_855137_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/f51a547a-90c2-42ef-8196-15c2575da80c_isr_855137_37.stderr parsl.dataflow.dflow DEBUG: Task 1096 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1096 submitted for App characterizeImage, waiting on task 1095 parsl.dataflow.dflow DEBUG: Task 1096 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1096 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1097 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1097 submitted for App calibrate, waiting on task 1096 parsl.dataflow.dflow DEBUG: Task 1097 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1097 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1098 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1098 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1098 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1098 has memoization hash e1d944ad77d97029a430694d7226d9b2 parsl.dataflow.memoization INFO: Task 1098 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa453918a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1098 try 0 launched on executor multi with executor id 367 parsl.dataflow.dflow INFO: Standard output for task 1098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/cec5c57e-142c-4c54-9461-bed07c9cbcfb_isr_855042_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/cec5c57e-142c-4c54-9461-bed07c9cbcfb_isr_855042_46.stderr parsl.dataflow.dflow DEBUG: Task 1099 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1099 submitted for App characterizeImage, waiting on task 1098 parsl.dataflow.dflow DEBUG: Task 1099 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1099 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1100 submitted for App calibrate, waiting on task 1099 parsl.dataflow.dflow DEBUG: Task 1100 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1100 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1101 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1101 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1101 has memoization hash 52753e7e8068cd9c1805481b6f29659b parsl.dataflow.memoization INFO: Task 1101 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45391940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1101 try 0 launched on executor multi with executor id 368 parsl.dataflow.dflow INFO: Standard output for task 1101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/844c6b77-1477-41e0-a3ef-f0a71cf780b3_isr_855136_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/844c6b77-1477-41e0-a3ef-f0a71cf780b3_isr_855136_18.stderr parsl.dataflow.dflow DEBUG: Task 1102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1102 submitted for App characterizeImage, waiting on task 1101 parsl.dataflow.dflow DEBUG: Task 1102 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1102 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1103 submitted for App calibrate, waiting on task 1102 parsl.dataflow.dflow DEBUG: Task 1103 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1103 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1104 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1104 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1104 has memoization hash 248b54f520dbcca6fbf5a2c90c743c3f parsl.dataflow.memoization INFO: Task 1104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45391e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1104 try 0 launched on executor multi with executor id 369 parsl.dataflow.dflow INFO: Standard output for task 1104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/d941a126-9b8f-4e8b-bc08-2617a88bab4d_isr_855138_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/d941a126-9b8f-4e8b-bc08-2617a88bab4d_isr_855138_23.stderr parsl.dataflow.dflow DEBUG: Task 1105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1105 submitted for App characterizeImage, waiting on task 1104 parsl.dataflow.dflow DEBUG: Task 1105 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1105 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1106 submitted for App calibrate, waiting on task 1105 parsl.dataflow.dflow DEBUG: Task 1106 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1106 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1107 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1107 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1107 has memoization hash d1b9678b5dd284437c138057f12e3d32 parsl.dataflow.memoization INFO: Task 1107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45392660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1107 try 0 launched on executor multi with executor id 370 parsl.dataflow.dflow INFO: Standard output for task 1107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/d96af26f-f987-4dc0-a77e-f77a676a61b6_isr_855138_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/d96af26f-f987-4dc0-a77e-f77a676a61b6_isr_855138_15.stderr parsl.dataflow.dflow DEBUG: Task 1108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1108 submitted for App characterizeImage, waiting on task 1107 parsl.dataflow.dflow DEBUG: Task 1108 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1108 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1109 submitted for App calibrate, waiting on task 1108 parsl.dataflow.dflow DEBUG: Task 1109 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1109 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1110 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1110 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1110 has memoization hash 310e2334406b4255bcb1ce13fa325de7 parsl.dataflow.memoization INFO: Task 1110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa453922a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1110 try 0 launched on executor multi with executor id 371 parsl.dataflow.dflow INFO: Standard output for task 1110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/bef771fd-6c3a-49bb-b4a9-772db22c61e8_isr_855138_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/bef771fd-6c3a-49bb-b4a9-772db22c61e8_isr_855138_54.stderr parsl.dataflow.dflow DEBUG: Task 1111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1111 submitted for App characterizeImage, waiting on task 1110 parsl.dataflow.dflow DEBUG: Task 1111 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1111 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1112 submitted for App calibrate, waiting on task 1111 parsl.dataflow.dflow DEBUG: Task 1112 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1112 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1113 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1113 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1113 has memoization hash 5498c4883e37b29d42c8342da93bddb7 parsl.dataflow.memoization INFO: Task 1113 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa453934c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1113 try 0 launched on executor multi with executor id 372 parsl.dataflow.dflow INFO: Standard output for task 1113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/5f1ec6d8-c8ab-4c57-a70d-1dfb67fafd21_isr_855137_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/5f1ec6d8-c8ab-4c57-a70d-1dfb67fafd21_isr_855137_56.stderr parsl.dataflow.dflow DEBUG: Task 1114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1114 submitted for App characterizeImage, waiting on task 1113 parsl.dataflow.dflow DEBUG: Task 1114 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1114 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1115 submitted for App calibrate, waiting on task 1114 parsl.dataflow.dflow DEBUG: Task 1115 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1115 has outstanding dependencies, so launch_if_ready skipping 2024-03-08 07:15:15 proc_lsst.multi:170 [INFO] [multi] job_id 16912292 proc_lsst.multi INFO: [multi] job_id 16912292 2024-03-08 07:15:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-08 07:15:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:15 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16912292 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16912292 parsl.executors.status_handling DEBUG: Launched block 10 on executor multi with job ID 16912292 parsl.executors.status_handling INFO: Allocated block ID 11 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-11 2024-03-08 07:15:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 2024-03-08 07:15:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-08 07:15:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:15 proc_lsst.multi:170 [INFO] [multi] job_id 16912293 proc_lsst.multi INFO: [multi] job_id 16912293 2024-03-08 07:15:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-08 07:15:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:15 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16912293 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16912293 parsl.executors.status_handling DEBUG: Launched block 11 on executor multi with job ID 16912293 parsl.executors.status_handling INFO: Allocated block ID 12 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-12 2024-03-08 07:15:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 2024-03-08 07:15:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-08 07:15:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:15 proc_lsst.multi:170 [INFO] [multi] job_id 16912294 proc_lsst.multi INFO: [multi] job_id 16912294 2024-03-08 07:15:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-08 07:15:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:15 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16912294 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16912294 parsl.executors.status_handling DEBUG: Launched block 12 on executor multi with job ID 16912294 parsl.executors.status_handling INFO: Allocated block ID 13 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-13 2024-03-08 07:15:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 2024-03-08 07:15:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-08 07:15:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:15 proc_lsst.multi:170 [INFO] [multi] job_id 16912295 proc_lsst.multi INFO: [multi] job_id 16912295 2024-03-08 07:15:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-08 07:15:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:15 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16912295 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16912295 parsl.executors.status_handling DEBUG: Launched block 13 on executor multi with job ID 16912295 parsl.executors.status_handling INFO: Allocated block ID 14 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-14 2024-03-08 07:15:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 2024-03-08 07:15:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-08 07:15:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:16 proc_lsst.multi:170 [INFO] [multi] job_id 16912296 proc_lsst.multi INFO: [multi] job_id 16912296 2024-03-08 07:15:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-08 07:15:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:16 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16912296 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16912296 parsl.executors.status_handling DEBUG: Launched block 14 on executor multi with job ID 16912296 parsl.executors.status_handling INFO: Allocated block ID 15 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-15 2024-03-08 07:15:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 2024-03-08 07:15:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-08 07:15:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:16 proc_lsst.multi:170 [INFO] [multi] job_id 16912297 proc_lsst.multi INFO: [multi] job_id 16912297 2024-03-08 07:15:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-08 07:15:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:16 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16912297 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16912297 parsl.executors.status_handling DEBUG: Launched block 15 on executor multi with job ID 16912297 parsl.executors.status_handling INFO: Allocated block ID 16 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-16 2024-03-08 07:15:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 2024-03-08 07:15:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-08 07:15:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:16 proc_lsst.multi:170 [INFO] [multi] job_id 16912298 proc_lsst.multi INFO: [multi] job_id 16912298 2024-03-08 07:15:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-08 07:15:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:16 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16912298 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16912298 parsl.executors.status_handling DEBUG: Launched block 16 on executor multi with job ID 16912298 parsl.executors.status_handling INFO: Allocated block ID 17 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-17 2024-03-08 07:15:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 2024-03-08 07:15:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-08 07:15:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:16 proc_lsst.multi:170 [INFO] [multi] job_id 16912299 proc_lsst.multi INFO: [multi] job_id 16912299 2024-03-08 07:15:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-08 07:15:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:16 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16912299 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16912299 parsl.executors.status_handling DEBUG: Launched block 17 on executor multi with job ID 16912299 parsl.executors.status_handling INFO: Allocated block ID 18 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-18 2024-03-08 07:15:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 2024-03-08 07:15:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-08 07:15:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:17 proc_lsst.multi:170 [INFO] [multi] job_id 16912300 proc_lsst.multi INFO: [multi] job_id 16912300 2024-03-08 07:15:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-08 07:15:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:17 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16912300 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16912300 parsl.executors.status_handling DEBUG: Launched block 18 on executor multi with job ID 16912300 parsl.executors.status_handling INFO: Allocated block ID 19 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-19 2024-03-08 07:15:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 2024-03-08 07:15:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-08 07:15:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:17 proc_lsst.multi:170 [INFO] [multi] job_id 16912301 proc_lsst.multi INFO: [multi] job_id 16912301 2024-03-08 07:15:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-08 07:15:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:17 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16912301 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16912301 parsl.executors.status_handling DEBUG: Launched block 19 on executor multi with job ID 16912301 parsl.executors.status_handling INFO: Allocated block ID 20 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-20 2024-03-08 07:15:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 2024-03-08 07:15:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-08 07:15:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:17 proc_lsst.multi:170 [INFO] [multi] job_id 16912302 proc_lsst.multi INFO: [multi] job_id 16912302 2024-03-08 07:15:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 07:15:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:17 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16912302 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16912302 parsl.executors.status_handling DEBUG: Launched block 20 on executor multi with job ID 16912302 parsl.executors.status_handling INFO: Allocated block ID 21 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-21 2024-03-08 07:15:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 2024-03-08 07:15:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 07:15:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:18 proc_lsst.multi:170 [INFO] [multi] job_id 16912303 proc_lsst.multi INFO: [multi] job_id 16912303 2024-03-08 07:15:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-08 07:15:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:18 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16912303 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16912303 parsl.executors.status_handling DEBUG: Launched block 21 on executor multi with job ID 16912303 parsl.executors.status_handling INFO: Allocated block ID 22 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-22 2024-03-08 07:15:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 2024-03-08 07:15:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-08 07:15:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:18 proc_lsst.multi:170 [INFO] [multi] job_id 16912304 proc_lsst.multi INFO: [multi] job_id 16912304 2024-03-08 07:15:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-08 07:15:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:18 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16912304 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16912304 parsl.executors.status_handling DEBUG: Launched block 22 on executor multi with job ID 16912304 parsl.executors.status_handling INFO: Allocated block ID 23 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-23 2024-03-08 07:15:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 2024-03-08 07:15:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-08 07:15:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:18 proc_lsst.multi:170 [INFO] [multi] job_id 16912305 proc_lsst.multi INFO: [multi] job_id 16912305 2024-03-08 07:15:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-08 07:15:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:18 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16912305 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16912305 parsl.executors.status_handling DEBUG: Launched block 23 on executor multi with job ID 16912305 parsl.executors.status_handling INFO: Allocated block ID 24 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-24 2024-03-08 07:15:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 2024-03-08 07:15:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-08 07:15:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:18 proc_lsst.multi:170 [INFO] [multi] job_id 16912306 proc_lsst.multi INFO: [multi] job_id 16912306 2024-03-08 07:15:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-08 07:15:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:18 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16912306 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16912306 parsl.executors.status_handling DEBUG: Launched block 24 on executor multi with job ID 16912306 parsl.executors.status_handling INFO: Allocated block ID 25 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-25 2024-03-08 07:15:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 2024-03-08 07:15:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-08 07:15:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:18 proc_lsst.multi:170 [INFO] [multi] job_id 16912307 proc_lsst.multi INFO: [multi] job_id 16912307 2024-03-08 07:15:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-08 07:15:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:18 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16912307 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16912307 parsl.executors.status_handling DEBUG: Launched block 25 on executor multi with job ID 16912307 parsl.executors.status_handling INFO: Allocated block ID 26 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-26 2024-03-08 07:15:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 2024-03-08 07:15:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-08 07:15:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:18 proc_lsst.multi:170 [INFO] [multi] job_id 16912308 proc_lsst.multi INFO: [multi] job_id 16912308 2024-03-08 07:15:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-08 07:15:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:18 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16912308 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16912308 parsl.executors.status_handling DEBUG: Launched block 26 on executor multi with job ID 16912308 parsl.executors.status_handling INFO: Allocated block ID 27 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-27 2024-03-08 07:15:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 2024-03-08 07:15:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-08 07:15:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:18 proc_lsst.multi:170 [INFO] [multi] job_id 16912309 proc_lsst.multi INFO: [multi] job_id 16912309 2024-03-08 07:15:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-08 07:15:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:18 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16912309 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16912309 parsl.executors.status_handling DEBUG: Launched block 27 on executor multi with job ID 16912309 parsl.executors.status_handling INFO: Allocated block ID 28 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-28 2024-03-08 07:15:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 2024-03-08 07:15:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-08 07:15:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:18 proc_lsst.multi:170 [INFO] [multi] job_id 16912310 proc_lsst.multi INFO: [multi] job_id 16912310 2024-03-08 07:15:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-08 07:15:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:18 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16912310 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16912310 parsl.executors.status_handling DEBUG: Launched block 28 on executor multi with job ID 16912310 parsl.executors.status_handling INFO: Allocated block ID 29 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-29 2024-03-08 07:15:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 2024-03-08 07:15:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-08 07:15:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:19 proc_lsst.multi:170 [INFO] [multi] job_id 16912311 proc_lsst.multi INFO: [multi] job_id 16912311 2024-03-08 07:15:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-08 07:15:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:19 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16912311 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16912311 parsl.executors.status_handling DEBUG: Launched block 29 on executor multi with job ID 16912311 parsl.executors.status_handling INFO: Allocated block ID 30 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-30 2024-03-08 07:15:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 2024-03-08 07:15:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-08 07:15:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:19 proc_lsst.multi:170 [INFO] [multi] job_id 16912312 proc_lsst.multi INFO: [multi] job_id 16912312 2024-03-08 07:15:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:19 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16912312 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16912312 parsl.executors.status_handling DEBUG: Launched block 30 on executor multi with job ID 16912312 parsl.executors.status_handling INFO: Allocated block ID 31 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-31 2024-03-08 07:15:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 2024-03-08 07:15:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:19 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-08 07:15:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:19 proc_lsst.multi:170 [INFO] [multi] job_id 16912313 proc_lsst.multi INFO: [multi] job_id 16912313 2024-03-08 07:15:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:19 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16912313 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16912313 parsl.executors.status_handling DEBUG: Launched block 31 on executor multi with job ID 16912313 parsl.executors.status_handling INFO: Allocated block ID 32 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-32 2024-03-08 07:15:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 2024-03-08 07:15:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:19 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:19 proc_lsst.multi:170 [INFO] [multi] job_id 16912314 proc_lsst.multi INFO: [multi] job_id 16912314 2024-03-08 07:15:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-08 07:15:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:19 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16912314 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16912314 parsl.executors.status_handling DEBUG: Launched block 32 on executor multi with job ID 16912314 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 33, active_tasks = 372 parsl.dataflow.strategy DEBUG: Executor multi has 372 active tasks, 1/32 running/pending blocks, and 8 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 33 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 60 more blocks parsl.executors.status_handling INFO: Scaling out by 60 blocks parsl.executors.status_handling INFO: Allocated block ID 33 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-33 2024-03-08 07:15:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 2024-03-08 07:15:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:19 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-08 07:15:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:19 proc_lsst.multi:170 [INFO] [multi] job_id 16912315 proc_lsst.multi INFO: [multi] job_id 16912315 2024-03-08 07:15:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-08 07:15:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:19 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16912315 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16912315 parsl.executors.status_handling DEBUG: Launched block 33 on executor multi with job ID 16912315 parsl.executors.status_handling INFO: Allocated block ID 34 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-34 2024-03-08 07:15:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 2024-03-08 07:15:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:19 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-08 07:15:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:20 proc_lsst.multi:170 [INFO] [multi] job_id 16912316 proc_lsst.multi INFO: [multi] job_id 16912316 2024-03-08 07:15:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-08 07:15:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:20 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16912316 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16912316 parsl.executors.status_handling DEBUG: Launched block 34 on executor multi with job ID 16912316 parsl.executors.status_handling INFO: Allocated block ID 35 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-35 2024-03-08 07:15:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 2024-03-08 07:15:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-08 07:15:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:20 proc_lsst.multi:170 [INFO] [multi] job_id 16912317 proc_lsst.multi INFO: [multi] job_id 16912317 2024-03-08 07:15:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:20 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16912317 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16912317 parsl.executors.status_handling DEBUG: Launched block 35 on executor multi with job ID 16912317 parsl.executors.status_handling INFO: Allocated block ID 36 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-36 2024-03-08 07:15:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 2024-03-08 07:15:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:20 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-08 07:15:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:20 proc_lsst.multi:170 [INFO] [multi] job_id 16912318 proc_lsst.multi INFO: [multi] job_id 16912318 2024-03-08 07:15:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:20 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16912318 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16912318 parsl.executors.status_handling DEBUG: Launched block 36 on executor multi with job ID 16912318 parsl.executors.status_handling INFO: Allocated block ID 37 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-37 2024-03-08 07:15:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 2024-03-08 07:15:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:20 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:20 proc_lsst.multi:170 [INFO] [multi] job_id 16912319 proc_lsst.multi INFO: [multi] job_id 16912319 2024-03-08 07:15:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-08 07:15:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:20 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16912319 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16912319 parsl.executors.status_handling DEBUG: Launched block 37 on executor multi with job ID 16912319 parsl.executors.status_handling INFO: Allocated block ID 38 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-38 2024-03-08 07:15:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 2024-03-08 07:15:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:20 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-08 07:15:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:20 proc_lsst.multi:170 [INFO] [multi] job_id 16912320 proc_lsst.multi INFO: [multi] job_id 16912320 2024-03-08 07:15:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-08 07:15:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:20 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16912320 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16912320 parsl.executors.status_handling DEBUG: Launched block 38 on executor multi with job ID 16912320 parsl.executors.status_handling INFO: Allocated block ID 39 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-39 2024-03-08 07:15:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 2024-03-08 07:15:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:20 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-08 07:15:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:20 proc_lsst.multi:170 [INFO] [multi] job_id 16912321 proc_lsst.multi INFO: [multi] job_id 16912321 2024-03-08 07:15:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-08 07:15:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:20 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16912321 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16912321 parsl.executors.status_handling DEBUG: Launched block 39 on executor multi with job ID 16912321 parsl.executors.status_handling INFO: Allocated block ID 40 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-40 2024-03-08 07:15:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 2024-03-08 07:15:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:20 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-08 07:15:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:21 proc_lsst.multi:170 [INFO] [multi] job_id 16912322 proc_lsst.multi INFO: [multi] job_id 16912322 2024-03-08 07:15:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:21 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16912322 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16912322 parsl.executors.status_handling DEBUG: Launched block 40 on executor multi with job ID 16912322 parsl.executors.status_handling INFO: Allocated block ID 41 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-41 2024-03-08 07:15:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 2024-03-08 07:15:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:21 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:21 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:21 proc_lsst.multi:170 [INFO] [multi] job_id 16912323 proc_lsst.multi INFO: [multi] job_id 16912323 2024-03-08 07:15:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-08 07:15:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:21 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16912323 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16912323 parsl.executors.status_handling DEBUG: Launched block 41 on executor multi with job ID 16912323 parsl.executors.status_handling INFO: Allocated block ID 42 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-42 2024-03-08 07:15:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 2024-03-08 07:15:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:21 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:21 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-08 07:15:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:21 proc_lsst.multi:170 [INFO] [multi] job_id 16912324 proc_lsst.multi INFO: [multi] job_id 16912324 2024-03-08 07:15:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-08 07:15:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:21 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16912324 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16912324 parsl.executors.status_handling DEBUG: Launched block 42 on executor multi with job ID 16912324 parsl.executors.status_handling INFO: Allocated block ID 43 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-43 2024-03-08 07:15:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 2024-03-08 07:15:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:21 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:21 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-08 07:15:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:22 proc_lsst.multi:170 [INFO] [multi] job_id 16912325 proc_lsst.multi INFO: [multi] job_id 16912325 2024-03-08 07:15:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-08 07:15:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:22 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16912325 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16912325 parsl.executors.status_handling DEBUG: Launched block 43 on executor multi with job ID 16912325 parsl.executors.status_handling INFO: Allocated block ID 44 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-44 2024-03-08 07:15:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 2024-03-08 07:15:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:22 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:22 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-08 07:15:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:22 proc_lsst.multi:170 [INFO] [multi] job_id 16912326 proc_lsst.multi INFO: [multi] job_id 16912326 2024-03-08 07:15:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-08 07:15:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:22 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16912326 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16912326 parsl.executors.status_handling DEBUG: Launched block 44 on executor multi with job ID 16912326 parsl.executors.status_handling INFO: Allocated block ID 45 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-45 2024-03-08 07:15:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 2024-03-08 07:15:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:22 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:22 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-08 07:15:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:22 proc_lsst.multi:170 [INFO] [multi] job_id 16912327 proc_lsst.multi INFO: [multi] job_id 16912327 2024-03-08 07:15:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-08 07:15:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:22 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16912327 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16912327 parsl.executors.status_handling DEBUG: Launched block 45 on executor multi with job ID 16912327 parsl.executors.status_handling INFO: Allocated block ID 46 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-46 2024-03-08 07:15:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 2024-03-08 07:15:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:22 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:22 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-08 07:15:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:23 proc_lsst.multi:170 [INFO] [multi] job_id 16912328 proc_lsst.multi INFO: [multi] job_id 16912328 2024-03-08 07:15:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-08 07:15:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:23 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16912328 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16912328 parsl.executors.status_handling DEBUG: Launched block 46 on executor multi with job ID 16912328 parsl.executors.status_handling INFO: Allocated block ID 47 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-47 2024-03-08 07:15:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-47 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-47 2024-03-08 07:15:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:23 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:23 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-08 07:15:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:23 proc_lsst.multi:170 [INFO] [multi] job_id 16912329 proc_lsst.multi INFO: [multi] job_id 16912329 2024-03-08 07:15:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-08 07:15:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:23 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16912329 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16912329 parsl.executors.status_handling DEBUG: Launched block 47 on executor multi with job ID 16912329 parsl.executors.status_handling INFO: Allocated block ID 48 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-48 2024-03-08 07:15:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-48 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-48 2024-03-08 07:15:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:23 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:23 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-08 07:15:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:23 proc_lsst.multi:170 [INFO] [multi] job_id 16912330 proc_lsst.multi INFO: [multi] job_id 16912330 2024-03-08 07:15:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-08 07:15:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:23 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16912330 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16912330 parsl.executors.status_handling DEBUG: Launched block 48 on executor multi with job ID 16912330 parsl.executors.status_handling INFO: Allocated block ID 49 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-49 2024-03-08 07:15:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-49 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-49 2024-03-08 07:15:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:23 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:23 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-08 07:15:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:24 proc_lsst.multi:170 [INFO] [multi] job_id 16912331 proc_lsst.multi INFO: [multi] job_id 16912331 2024-03-08 07:15:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-08 07:15:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:24 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16912331 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16912331 parsl.executors.status_handling DEBUG: Launched block 49 on executor multi with job ID 16912331 parsl.executors.status_handling INFO: Allocated block ID 50 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-50 2024-03-08 07:15:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-50 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-50 2024-03-08 07:15:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:24 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:24 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-08 07:15:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:24 proc_lsst.multi:170 [INFO] [multi] job_id 16912332 proc_lsst.multi INFO: [multi] job_id 16912332 2024-03-08 07:15:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-08 07:15:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:24 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16912332 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16912332 parsl.executors.status_handling DEBUG: Launched block 50 on executor multi with job ID 16912332 parsl.executors.status_handling INFO: Allocated block ID 51 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-51 2024-03-08 07:15:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-51 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-51 2024-03-08 07:15:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:24 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:24 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-08 07:15:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:25 proc_lsst.multi:170 [INFO] [multi] job_id 16912333 proc_lsst.multi INFO: [multi] job_id 16912333 2024-03-08 07:15:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-08 07:15:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:25 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16912333 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16912333 parsl.executors.status_handling DEBUG: Launched block 51 on executor multi with job ID 16912333 parsl.executors.status_handling INFO: Allocated block ID 52 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-52 2024-03-08 07:15:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-52 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-52 2024-03-08 07:15:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:25 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:25 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-08 07:15:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:25 proc_lsst.multi:170 [INFO] [multi] job_id 16912334 proc_lsst.multi INFO: [multi] job_id 16912334 2024-03-08 07:15:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-08 07:15:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:25 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16912334 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16912334 parsl.executors.status_handling DEBUG: Launched block 52 on executor multi with job ID 16912334 parsl.executors.status_handling INFO: Allocated block ID 53 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-53 2024-03-08 07:15:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-53 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-53 2024-03-08 07:15:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:25 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:25 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-08 07:15:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:26 proc_lsst.multi:170 [INFO] [multi] job_id 16912335 proc_lsst.multi INFO: [multi] job_id 16912335 2024-03-08 07:15:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-08 07:15:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:26 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16912335 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16912335 parsl.executors.status_handling DEBUG: Launched block 53 on executor multi with job ID 16912335 parsl.executors.status_handling INFO: Allocated block ID 54 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-54 2024-03-08 07:15:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-54 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-54 2024-03-08 07:15:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:26 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:26 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-08 07:15:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 0 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b4244b5a6f29a54d9b8667c52fb483d with result from task 0 parsl.dataflow.dflow DEBUG: Task 0 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1 has memoization hash c53c74e0c2de213c82f364722bacb20f parsl.dataflow.memoization INFO: Task 1 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467b1ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-08 07:15:27 proc_lsst.multi:170 [INFO] [multi] job_id 16912336 proc_lsst.multi INFO: [multi] job_id 16912336 2024-03-08 07:15:27 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-08 07:15:27 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:27 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16912336 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16912336 parsl.executors.status_handling DEBUG: Launched block 54 on executor multi with job ID 16912336 parsl.executors.status_handling INFO: Allocated block ID 55 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-55 2024-03-08 07:15:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-55 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-55 2024-03-08 07:15:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:27 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:27 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-08 07:15:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1 try 0 launched on executor multi with executor id 373 parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/3a7f69fe-e49e-45fa-8476-ab705543b5a5_characterizeImage_855137_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/3a7f69fe-e49e-45fa-8476-ab705543b5a5_characterizeImage_855137_15.stderr parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/0e5243e1-505d-4a4e-9768-7671fca81598_isr_855137_15.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/0e5243e1-505d-4a4e-9768-7671fca81598_isr_855137_15.stderr 2024-03-08 07:15:27 proc_lsst.multi:170 [INFO] [multi] job_id 16912337 proc_lsst.multi INFO: [multi] job_id 16912337 2024-03-08 07:15:27 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 07:15:27 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:27 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16912337 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16912337 parsl.executors.status_handling DEBUG: Launched block 55 on executor multi with job ID 16912337 parsl.executors.status_handling INFO: Allocated block ID 56 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-56 2024-03-08 07:15:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-56 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-56 2024-03-08 07:15:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:27 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:27 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 07:15:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:27 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-08 07:15:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 3 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4cdb926e1410e01c1674c0d3ed22b44f with result from task 3 parsl.dataflow.dflow DEBUG: Task 3 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4 has memoization hash 80c50c9bc877bfd12dd664ee4cd46996 parsl.dataflow.memoization INFO: Task 4 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467b23e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4 try 0 launched on executor multi with executor id 374 parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/c88d74be-60a3-45bf-9997-ab1d22296af1_characterizeImage_855137_13.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/c88d74be-60a3-45bf-9997-ab1d22296af1_characterizeImage_855137_13.stderr parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/745c45ee-4ceb-4233-81e7-403a9d11165a_isr_855137_13.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/745c45ee-4ceb-4233-81e7-403a9d11165a_isr_855137_13.stderr 2024-03-08 07:15:28 proc_lsst.multi:170 [INFO] [multi] job_id 16912338 proc_lsst.multi INFO: [multi] job_id 16912338 2024-03-08 07:15:28 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:28 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:28 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16912338 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16912338 parsl.executors.status_handling DEBUG: Launched block 56 on executor multi with job ID 16912338 parsl.executors.status_handling INFO: Allocated block ID 57 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-57 2024-03-08 07:15:28 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-57 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-57 2024-03-08 07:15:28 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:28 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:28 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:28 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 07:15:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:28 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:28 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:28 proc_lsst.multi:170 [INFO] [multi] job_id 16912339 proc_lsst.multi INFO: [multi] job_id 16912339 2024-03-08 07:15:28 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-08 07:15:28 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:28 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16912339 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16912339 parsl.executors.status_handling DEBUG: Launched block 57 on executor multi with job ID 16912339 parsl.executors.status_handling INFO: Allocated block ID 58 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-58 2024-03-08 07:15:28 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-58 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-58 2024-03-08 07:15:28 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:28 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:28 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:28 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 07:15:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:28 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-08 07:15:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:28 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:29 proc_lsst.multi:170 [INFO] [multi] job_id 16912340 proc_lsst.multi INFO: [multi] job_id 16912340 2024-03-08 07:15:29 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-08 07:15:29 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:29 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16912340 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16912340 parsl.executors.status_handling DEBUG: Launched block 58 on executor multi with job ID 16912340 parsl.executors.status_handling INFO: Allocated block ID 59 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-59 2024-03-08 07:15:29 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-59 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-59 2024-03-08 07:15:29 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:29 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:29 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:29 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 07:15:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:29 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-08 07:15:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:29 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:29 proc_lsst.multi:170 [INFO] [multi] job_id 16912341 proc_lsst.multi INFO: [multi] job_id 16912341 2024-03-08 07:15:29 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-08 07:15:29 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:29 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16912341 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16912341 parsl.executors.status_handling DEBUG: Launched block 59 on executor multi with job ID 16912341 parsl.executors.status_handling INFO: Allocated block ID 60 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-60 2024-03-08 07:15:29 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-60 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-60 2024-03-08 07:15:29 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:29 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:29 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:29 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 07:15:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:29 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-08 07:15:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:29 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:30 proc_lsst.multi:170 [INFO] [multi] job_id 16912342 proc_lsst.multi INFO: [multi] job_id 16912342 2024-03-08 07:15:30 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:30 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:30 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16912342 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16912342 parsl.executors.status_handling DEBUG: Launched block 60 on executor multi with job ID 16912342 parsl.executors.status_handling INFO: Allocated block ID 61 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-61 2024-03-08 07:15:30 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-61 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-61 2024-03-08 07:15:30 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:30 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:30 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:30 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 07:15:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:30 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:30 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:31 proc_lsst.multi:170 [INFO] [multi] job_id 16912343 proc_lsst.multi INFO: [multi] job_id 16912343 2024-03-08 07:15:31 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-08 07:15:31 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:31 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16912343 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16912343 parsl.executors.status_handling DEBUG: Launched block 61 on executor multi with job ID 16912343 parsl.executors.status_handling INFO: Allocated block ID 62 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-62 2024-03-08 07:15:31 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-62 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-62 2024-03-08 07:15:31 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:31 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:31 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:31 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 07:15:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:31 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-08 07:15:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:31 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:32 proc_lsst.multi:170 [INFO] [multi] job_id 16912344 proc_lsst.multi INFO: [multi] job_id 16912344 2024-03-08 07:15:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-08 07:15:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:32 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16912344 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16912344 parsl.executors.status_handling DEBUG: Launched block 62 on executor multi with job ID 16912344 parsl.executors.status_handling INFO: Allocated block ID 63 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-63 2024-03-08 07:15:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-63 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-63 2024-03-08 07:15:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:32 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:32 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 07:15:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:32 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-08 07:15:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:32 proc_lsst.multi:170 [INFO] [multi] job_id 16912345 proc_lsst.multi INFO: [multi] job_id 16912345 2024-03-08 07:15:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-08 07:15:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:32 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16912345 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16912345 parsl.executors.status_handling DEBUG: Launched block 63 on executor multi with job ID 16912345 parsl.executors.status_handling INFO: Allocated block ID 64 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-64 2024-03-08 07:15:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-64 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-64 2024-03-08 07:15:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:32 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:32 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 07:15:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:32 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-08 07:15:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:32 proc_lsst.multi:170 [INFO] [multi] job_id 16912346 proc_lsst.multi INFO: [multi] job_id 16912346 2024-03-08 07:15:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-08 07:15:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:32 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16912346 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16912346 parsl.executors.status_handling DEBUG: Launched block 64 on executor multi with job ID 16912346 parsl.executors.status_handling INFO: Allocated block ID 65 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-65 2024-03-08 07:15:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-65 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-65 2024-03-08 07:15:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:32 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:32 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 07:15:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:32 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-08 07:15:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:33 proc_lsst.multi:170 [INFO] [multi] job_id 16912347 proc_lsst.multi INFO: [multi] job_id 16912347 2024-03-08 07:15:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-08 07:15:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:33 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16912347 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16912347 parsl.executors.status_handling DEBUG: Launched block 65 on executor multi with job ID 16912347 parsl.executors.status_handling INFO: Allocated block ID 66 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-66 2024-03-08 07:15:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-66 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-66 2024-03-08 07:15:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:33 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:33 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 07:15:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:33 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-08 07:15:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:33 proc_lsst.multi:170 [INFO] [multi] job_id 16912348 proc_lsst.multi INFO: [multi] job_id 16912348 2024-03-08 07:15:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-08 07:15:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:33 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16912348 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16912348 parsl.executors.status_handling DEBUG: Launched block 66 on executor multi with job ID 16912348 parsl.executors.status_handling INFO: Allocated block ID 67 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-67 2024-03-08 07:15:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-67 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-67 2024-03-08 07:15:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:33 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:33 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 07:15:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:33 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-08 07:15:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:33 proc_lsst.multi:170 [INFO] [multi] job_id 16912349 proc_lsst.multi INFO: [multi] job_id 16912349 2024-03-08 07:15:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-08 07:15:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:33 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16912349 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16912349 parsl.executors.status_handling DEBUG: Launched block 67 on executor multi with job ID 16912349 parsl.executors.status_handling INFO: Allocated block ID 68 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-68 2024-03-08 07:15:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-68 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-68 2024-03-08 07:15:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:33 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:33 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 07:15:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:33 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-08 07:15:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:34 proc_lsst.multi:170 [INFO] [multi] job_id 16912350 proc_lsst.multi INFO: [multi] job_id 16912350 2024-03-08 07:15:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-08 07:15:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:34 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16912350 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16912350 parsl.executors.status_handling DEBUG: Launched block 68 on executor multi with job ID 16912350 parsl.executors.status_handling INFO: Allocated block ID 69 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-69 2024-03-08 07:15:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-69 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-69 2024-03-08 07:15:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:34 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:34 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 07:15:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:34 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-08 07:15:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:34 proc_lsst.multi:170 [INFO] [multi] job_id 16912351 proc_lsst.multi INFO: [multi] job_id 16912351 2024-03-08 07:15:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-08 07:15:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:34 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16912351 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16912351 parsl.executors.status_handling DEBUG: Launched block 69 on executor multi with job ID 16912351 parsl.executors.status_handling INFO: Allocated block ID 70 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-70 2024-03-08 07:15:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-70 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-70 2024-03-08 07:15:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:34 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:34 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 07:15:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:34 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-08 07:15:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:34 proc_lsst.multi:170 [INFO] [multi] job_id 16912352 proc_lsst.multi INFO: [multi] job_id 16912352 2024-03-08 07:15:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-08 07:15:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:34 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16912352 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16912352 parsl.executors.status_handling DEBUG: Launched block 70 on executor multi with job ID 16912352 parsl.executors.status_handling INFO: Allocated block ID 71 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-71 2024-03-08 07:15:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-71 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-71 2024-03-08 07:15:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:34 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:34 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 07:15:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:34 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-08 07:15:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:35 proc_lsst.multi:170 [INFO] [multi] job_id 16912353 proc_lsst.multi INFO: [multi] job_id 16912353 2024-03-08 07:15:35 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-08 07:15:35 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:35 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16912353 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16912353 parsl.executors.status_handling DEBUG: Launched block 71 on executor multi with job ID 16912353 parsl.executors.status_handling INFO: Allocated block ID 72 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-72 2024-03-08 07:15:35 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-72 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-72 2024-03-08 07:15:35 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:35 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:35 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:35 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 07:15:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:35 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-08 07:15:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:35 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 54 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c504d2a58fcc16d018459ddd0b1e3b81 with result from task 54 parsl.dataflow.dflow DEBUG: Task 54 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 55 has memoization hash d071f37d13ed32c008a7064673b7d58c parsl.dataflow.memoization INFO: Task 55 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d1da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 55 try 0 launched on executor multi with executor id 375 parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/9301d035-4d33-4249-aba1-7e5b01ae9e91_characterizeImage_855044_8.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/9301d035-4d33-4249-aba1-7e5b01ae9e91_characterizeImage_855044_8.stderr parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/6fa0f901-3f62-477c-870c-c34c9e2c6c53_isr_855044_8.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/6fa0f901-3f62-477c-870c-c34c9e2c6c53_isr_855044_8.stderr 2024-03-08 07:15:36 proc_lsst.multi:170 [INFO] [multi] job_id 16912354 proc_lsst.multi INFO: [multi] job_id 16912354 2024-03-08 07:15:36 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-08 07:15:36 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:36 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16912354 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16912354 parsl.executors.status_handling DEBUG: Launched block 72 on executor multi with job ID 16912354 parsl.executors.status_handling INFO: Allocated block ID 73 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-73 2024-03-08 07:15:36 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-73 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-73 2024-03-08 07:15:36 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:36 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:36 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:36 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 07:15:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:36 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-08 07:15:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:36 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 57 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b8a977ca53916779e3361d393756b7f with result from task 57 parsl.dataflow.dflow DEBUG: Task 57 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 58 has memoization hash 6f072ca7408d5c7fc82e7e395cd36492 parsl.dataflow.memoization INFO: Task 58 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d20c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 58 try 0 launched on executor multi with executor id 376 parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/d6cb7c98-9da1-444d-bacb-75afb618c81a_characterizeImage_855136_62.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/d6cb7c98-9da1-444d-bacb-75afb618c81a_characterizeImage_855136_62.stderr parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/f33248ce-785c-4bec-93c4-5a8ff29440c1_isr_855136_62.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/f33248ce-785c-4bec-93c4-5a8ff29440c1_isr_855136_62.stderr 2024-03-08 07:15:38 proc_lsst.multi:170 [INFO] [multi] job_id 16912355 proc_lsst.multi INFO: [multi] job_id 16912355 2024-03-08 07:15:38 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-08 07:15:38 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:38 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16912355 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16912355 parsl.executors.status_handling DEBUG: Launched block 73 on executor multi with job ID 16912355 parsl.executors.status_handling INFO: Allocated block ID 74 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-74 2024-03-08 07:15:38 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-74 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-74 2024-03-08 07:15:38 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:38 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:38 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:38 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 07:15:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:38 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-08 07:15:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:38 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 84 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b453e09970d7ebe30fbf33c5675e12df with result from task 84 parsl.dataflow.dflow DEBUG: Task 84 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 85 has memoization hash c118e3119241cb3db0960ff8e09fe49f parsl.dataflow.memoization INFO: Task 85 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d3ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 85 try 0 launched on executor multi with executor id 377 parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/54dc39bb-bdc0-4e4b-993e-2f1e038550f6_characterizeImage_855137_21.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/54dc39bb-bdc0-4e4b-993e-2f1e038550f6_characterizeImage_855137_21.stderr parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/3f2772ca-3a2f-4301-a402-8d4fe3c4707f_isr_855137_21.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/3f2772ca-3a2f-4301-a402-8d4fe3c4707f_isr_855137_21.stderr parsl.dataflow.dflow INFO: Task 81 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4ea801dae1c29c6043469a1b4f64de2 with result from task 81 parsl.dataflow.dflow DEBUG: Task 81 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 82 has memoization hash 88dc7bb497c36290e27d79d0fb453e8f parsl.dataflow.memoization INFO: Task 82 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d39c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 82 try 0 launched on executor multi with executor id 378 parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/9c3a29e5-0fba-46e2-aa95-5515ba146a65_characterizeImage_855138_8.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/9c3a29e5-0fba-46e2-aa95-5515ba146a65_characterizeImage_855138_8.stderr parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/e6fe17cf-a952-4a1f-81c0-0bae54e15f5b_isr_855138_8.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/e6fe17cf-a952-4a1f-81c0-0bae54e15f5b_isr_855138_8.stderr parsl.dataflow.dflow INFO: Task 93 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5cf6a48f232472395d7640388c1d1d56 with result from task 93 parsl.dataflow.dflow DEBUG: Task 93 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 94 has memoization hash c1e7393065d64717fc20b42df238730f parsl.dataflow.memoization INFO: Task 94 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f4680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 94 try 0 launched on executor multi with executor id 379 parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/ceb0c9a2-7712-48b9-9626-5acce65e4ad1_characterizeImage_855137_30.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/ceb0c9a2-7712-48b9-9626-5acce65e4ad1_characterizeImage_855137_30.stderr parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/f0a6b92b-ae86-46bb-bf34-4b350b752a97_isr_855137_30.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/f0a6b92b-ae86-46bb-bf34-4b350b752a97_isr_855137_30.stderr parsl.dataflow.dflow INFO: Task 45 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6399b24fff620748348edc0e8030088f with result from task 45 parsl.dataflow.dflow DEBUG: Task 45 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 46 has memoization hash 3f0c6239bea39b99ae448343c1e760c0 parsl.dataflow.memoization INFO: Task 46 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d1440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 46 try 0 launched on executor multi with executor id 380 parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/200eeb73-9f08-4d6a-a7d6-ab66fb193ebd_characterizeImage_855138_32.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/200eeb73-9f08-4d6a-a7d6-ab66fb193ebd_characterizeImage_855138_32.stderr parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/9af661ba-a4a5-4c75-bef9-743e35dc0c79_isr_855138_32.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/9af661ba-a4a5-4c75-bef9-743e35dc0c79_isr_855138_32.stderr parsl.dataflow.dflow INFO: Task 6 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5cb21205d230da60251cb2a4ac46cf73 with result from task 6 parsl.dataflow.dflow DEBUG: Task 6 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 7 has memoization hash a2d9112c3c8e05ec8b561d32911f3d6b parsl.dataflow.memoization INFO: Task 7 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467b28e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 7 try 0 launched on executor multi with executor id 381 parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/79bf686e-fa33-461d-b1ea-0bed46f8f9b4_characterizeImage_855044_41.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/79bf686e-fa33-461d-b1ea-0bed46f8f9b4_characterizeImage_855044_41.stderr parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/7ceaf897-72f4-42d8-9230-00bb5c51213d_isr_855044_41.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/7ceaf897-72f4-42d8-9230-00bb5c51213d_isr_855044_41.stderr parsl.dataflow.dflow INFO: Task 27 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d2b46b4b4bf86fabbdfd37636d8a1e38 with result from task 27 parsl.dataflow.dflow DEBUG: Task 27 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 28 has memoization hash 0d88c35be37c6f191d0a09d9df87062e parsl.dataflow.memoization INFO: Task 28 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467b3b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 28 try 0 launched on executor multi with executor id 382 parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/b2536c1b-d8be-4dce-a22e-3b2aa66a65b0_characterizeImage_855137_19.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/b2536c1b-d8be-4dce-a22e-3b2aa66a65b0_characterizeImage_855137_19.stderr parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/eb386060-bf2a-4275-ad0b-d9d0624dac5c_isr_855137_19.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/eb386060-bf2a-4275-ad0b-d9d0624dac5c_isr_855137_19.stderr 2024-03-08 07:15:40 proc_lsst.multi:170 [INFO] [multi] job_id 16912356 proc_lsst.multi INFO: [multi] job_id 16912356 2024-03-08 07:15:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-08 07:15:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:40 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16912356 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16912356 parsl.executors.status_handling DEBUG: Launched block 74 on executor multi with job ID 16912356 parsl.executors.status_handling INFO: Allocated block ID 75 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-75 2024-03-08 07:15:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-75 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-75 2024-03-08 07:15:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:40 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:40 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 07:15:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:40 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-08 07:15:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 51 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 233538819bbe48102d9ee02dbe0f6a77 with result from task 51 parsl.dataflow.dflow DEBUG: Task 51 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 52 has memoization hash 00b363912a81b2f885cc9017b26ceb45 parsl.dataflow.memoization INFO: Task 52 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d1a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 52 try 0 launched on executor multi with executor id 383 parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/ef798390-5f38-4196-826a-0e634d07ecf7_characterizeImage_855137_34.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/ef798390-5f38-4196-826a-0e634d07ecf7_characterizeImage_855137_34.stderr parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/3a9c8142-aaed-4475-9d76-e58170699c95_isr_855137_34.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/3a9c8142-aaed-4475-9d76-e58170699c95_isr_855137_34.stderr parsl.dataflow.dflow INFO: Task 24 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01250d59c1f6255c5477d0c604c3b3a2 with result from task 24 parsl.dataflow.dflow DEBUG: Task 24 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 25 has memoization hash 27de2e521cec85af3a4d51e8f44cf038 parsl.dataflow.memoization INFO: Task 25 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467b3c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 25 try 0 launched on executor multi with executor id 384 parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/b332937c-9ff4-4ee8-b096-8591a8b80ae6_characterizeImage_855044_22.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/b332937c-9ff4-4ee8-b096-8591a8b80ae6_characterizeImage_855044_22.stderr parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/c1c7848a-4ad2-4eb1-a232-9aaab8993286_isr_855044_22.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/c1c7848a-4ad2-4eb1-a232-9aaab8993286_isr_855044_22.stderr parsl.dataflow.dflow INFO: Task 105 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a9ccf97efa854bbed3a7185bc1dfffa with result from task 105 parsl.dataflow.dflow DEBUG: Task 105 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 106 has memoization hash 4cb89669799b6056e39b1c0d91523925 parsl.dataflow.memoization INFO: Task 106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f5300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 106 try 0 launched on executor multi with executor id 385 parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/52ddafda-9886-4e57-9fc8-7d8b140e300d_characterizeImage_855042_14.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/52ddafda-9886-4e57-9fc8-7d8b140e300d_characterizeImage_855042_14.stderr parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/5455d6c8-0ac0-427e-b2a7-daecfb54a4c4_isr_855042_14.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/5455d6c8-0ac0-427e-b2a7-daecfb54a4c4_isr_855042_14.stderr parsl.dataflow.dflow INFO: Task 96 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 533ff86ca8c7fe6ea042948f3ccb7896 with result from task 96 parsl.dataflow.dflow DEBUG: Task 96 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 97 has memoization hash 3d28ac0e9240d793e07c6d97ffe98b68 parsl.dataflow.memoization INFO: Task 97 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f49a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 97 try 0 launched on executor multi with executor id 386 parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/65524315-6250-4b13-baeb-6b3881e6ffeb_characterizeImage_855042_55.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/65524315-6250-4b13-baeb-6b3881e6ffeb_characterizeImage_855042_55.stderr parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/1cee0486-f8b4-4aa8-926b-f243c2496b6c_isr_855042_55.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/1cee0486-f8b4-4aa8-926b-f243c2496b6c_isr_855042_55.stderr parsl.dataflow.dflow INFO: Task 66 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf70f7ea26d1e5de592f964252960bf3 with result from task 66 parsl.dataflow.dflow DEBUG: Task 66 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 67 has memoization hash 13f1cb49a919c3c87b6c7bff3d2ca7c0 parsl.dataflow.memoization INFO: Task 67 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d2a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 67 try 0 launched on executor multi with executor id 387 parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/10c17075-adc0-44f6-b018-0f25128bc1ad_characterizeImage_855136_34.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/10c17075-adc0-44f6-b018-0f25128bc1ad_characterizeImage_855136_34.stderr parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/5fa47311-04d2-4ef6-8ca3-01de7906989d_isr_855136_34.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/5fa47311-04d2-4ef6-8ca3-01de7906989d_isr_855136_34.stderr parsl.dataflow.dflow INFO: Task 63 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c5b81a965d45d625761114c031863d4 with result from task 63 parsl.dataflow.dflow DEBUG: Task 63 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 64 has memoization hash 575c44743cc37efc60a08afd0388548d parsl.dataflow.memoization INFO: Task 64 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d2700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 64 try 0 launched on executor multi with executor id 388 parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/2c763691-9176-48d0-a610-fb07257ee159_characterizeImage_855137_31.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/2c763691-9176-48d0-a610-fb07257ee159_characterizeImage_855137_31.stderr parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/50ccaf69-3436-4c00-a666-4b7c4ff8004f_isr_855137_31.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/50ccaf69-3436-4c00-a666-4b7c4ff8004f_isr_855137_31.stderr parsl.dataflow.dflow INFO: Task 90 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a1c566a1d375d9f83e7a737f60bd1ae with result from task 90 parsl.dataflow.dflow DEBUG: Task 90 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 91 has memoization hash 75c64e3911097d98d18c83d5ff9534ae parsl.dataflow.memoization INFO: Task 91 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f4360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 91 try 0 launched on executor multi with executor id 389 parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/5a73d06e-924a-40e1-a339-a4d4db7617bb_characterizeImage_855043_19.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/5a73d06e-924a-40e1-a339-a4d4db7617bb_characterizeImage_855043_19.stderr parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/40113a75-50a3-47d9-a673-a985635251d0_isr_855043_19.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/40113a75-50a3-47d9-a673-a985635251d0_isr_855043_19.stderr parsl.dataflow.dflow INFO: Task 87 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b620bc4535a906a8861e3cdc60779310 with result from task 87 parsl.dataflow.dflow DEBUG: Task 87 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 88 has memoization hash 90af7b201c7d32fbe0f483efea971dd3 parsl.dataflow.memoization INFO: Task 88 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d3e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 88 try 0 launched on executor multi with executor id 390 parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/912929fb-2d8b-4d6e-9832-458a1356f45b_characterizeImage_855044_28.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/912929fb-2d8b-4d6e-9832-458a1356f45b_characterizeImage_855044_28.stderr parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/c4155964-541b-41c4-acdc-216ffeec1a46_isr_855044_28.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/c4155964-541b-41c4-acdc-216ffeec1a46_isr_855044_28.stderr parsl.dataflow.dflow INFO: Task 36 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c58e72f23d919abeccdffed4b00ae68 with result from task 36 parsl.dataflow.dflow DEBUG: Task 36 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 37 has memoization hash b1d845a3183f8c3439435972138c0905 parsl.dataflow.memoization INFO: Task 37 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d09a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 37 try 0 launched on executor multi with executor id 391 parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/fbbb8cf4-a5ed-4e84-8876-4b6e31030e57_characterizeImage_855138_42.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/fbbb8cf4-a5ed-4e84-8876-4b6e31030e57_characterizeImage_855138_42.stderr parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/4cdd37f4-e5d4-448e-b0a3-017228c3e3c3_isr_855138_42.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/4cdd37f4-e5d4-448e-b0a3-017228c3e3c3_isr_855138_42.stderr parsl.dataflow.dflow INFO: Task 102 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d45c6969d95b8c59389a8cc01ef1dff with result from task 102 parsl.dataflow.dflow DEBUG: Task 102 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 103 has memoization hash db10c0304bf013606ba1e7996703d96d parsl.dataflow.memoization INFO: Task 103 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f4fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 103 try 0 launched on executor multi with executor id 392 parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/cb9145f4-6ba0-4665-9579-4e7e7d756055_characterizeImage_855136_3.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/cb9145f4-6ba0-4665-9579-4e7e7d756055_characterizeImage_855136_3.stderr parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/276fb408-af0e-44e2-954c-71b8032f89a2_isr_855136_3.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/276fb408-af0e-44e2-954c-71b8032f89a2_isr_855136_3.stderr parsl.dataflow.dflow INFO: Task 12 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3999b4c4c537a64ae39566a5d078f44a with result from task 12 parsl.dataflow.dflow DEBUG: Task 12 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 13 has memoization hash bffcae9bb77c0349839be3ec0d67e794 parsl.dataflow.memoization INFO: Task 13 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467b2fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 13 try 0 launched on executor multi with executor id 393 parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/0c42e1c5-eb02-40ea-bd49-1d8f5fa9c908_characterizeImage_855044_4.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/0c42e1c5-eb02-40ea-bd49-1d8f5fa9c908_characterizeImage_855044_4.stderr parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/53ae629e-8af0-490e-9ee2-cc3f2fcb179c_isr_855044_4.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/53ae629e-8af0-490e-9ee2-cc3f2fcb179c_isr_855044_4.stderr parsl.dataflow.dflow INFO: Task 15 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff4e539b7d6d06fce61bc4a27398f8ce with result from task 15 parsl.dataflow.dflow DEBUG: Task 15 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 16 has memoization hash 0a0d63fbd6f10d9cad2a14e70cbb4a37 parsl.dataflow.memoization INFO: Task 16 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467b32e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 16 try 0 launched on executor multi with executor id 394 parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/1d5302e0-40c6-4288-92ed-fb0d299d3f72_characterizeImage_855044_53.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/1d5302e0-40c6-4288-92ed-fb0d299d3f72_characterizeImage_855044_53.stderr parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/0f1cc0d6-1c97-4cf2-b716-0813e198509d_isr_855044_53.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/0f1cc0d6-1c97-4cf2-b716-0813e198509d_isr_855044_53.stderr parsl.dataflow.dflow INFO: Task 33 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8105b3ba42e27c05b314eb3fc5ee7be with result from task 33 parsl.dataflow.dflow DEBUG: Task 33 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 34 has memoization hash 2fa62125b2b362f1dbe20237dc66554e parsl.dataflow.memoization INFO: Task 34 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d0680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 34 try 0 launched on executor multi with executor id 395 parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/78263280-9d23-4733-aefc-ebeb93bdc068_characterizeImage_855137_7.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/78263280-9d23-4733-aefc-ebeb93bdc068_characterizeImage_855137_7.stderr parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/5e6c7cb6-c807-4de6-b790-4ed11872c0fd_isr_855137_7.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/5e6c7cb6-c807-4de6-b790-4ed11872c0fd_isr_855137_7.stderr parsl.dataflow.dflow INFO: Task 42 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da67f576481b02e70482a9142c01dd96 with result from task 42 parsl.dataflow.dflow DEBUG: Task 42 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 43 has memoization hash efd662c842b6ea42999b0e0bee126020 parsl.dataflow.memoization INFO: Task 43 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d1120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 43 try 0 launched on executor multi with executor id 396 parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/57ff7d87-8c9c-47a7-b148-25d7985035a6_characterizeImage_855136_35.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/57ff7d87-8c9c-47a7-b148-25d7985035a6_characterizeImage_855136_35.stderr parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/7675743d-1540-4938-82f0-19a9c8fafe6c_isr_855136_35.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/7675743d-1540-4938-82f0-19a9c8fafe6c_isr_855136_35.stderr parsl.dataflow.dflow INFO: Task 48 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 631f8aadff31ff50b8c953fefe5e37a6 with result from task 48 parsl.dataflow.dflow DEBUG: Task 48 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 49 has memoization hash d98ed34ad8268e57cc336c70f00f3309 parsl.dataflow.memoization INFO: Task 49 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d1760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 49 try 0 launched on executor multi with executor id 397 parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/e9474207-cb9d-4e42-aa92-bc3afc4336bf_characterizeImage_855138_55.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/e9474207-cb9d-4e42-aa92-bc3afc4336bf_characterizeImage_855138_55.stderr parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/7f4b07d1-b39c-4346-b540-fd4b2b4f3dfb_isr_855138_55.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/7f4b07d1-b39c-4346-b540-fd4b2b4f3dfb_isr_855138_55.stderr parsl.dataflow.dflow INFO: Task 69 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84c6d052106aa7324653b86b734a4231 with result from task 69 parsl.dataflow.dflow DEBUG: Task 69 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 70 has memoization hash 6b7e132571e3fc79cd71e18b821ebcc1 parsl.dataflow.memoization INFO: Task 70 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d2d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 70 try 0 launched on executor multi with executor id 398 parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/498ab39c-a1c9-4391-b8a0-aa2ce9b1733a_characterizeImage_855137_4.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/498ab39c-a1c9-4391-b8a0-aa2ce9b1733a_characterizeImage_855137_4.stderr parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/8ff78fda-7150-467b-8f20-a32b0bdc346c_isr_855137_4.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/8ff78fda-7150-467b-8f20-a32b0bdc346c_isr_855137_4.stderr parsl.dataflow.dflow INFO: Task 117 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 303da4a94976a4d70ea481d07ee17941 with result from task 117 parsl.dataflow.dflow DEBUG: Task 117 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 118 has memoization hash 90171a04074af488fac4cfc595aa2314 parsl.dataflow.memoization INFO: Task 118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f5f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-08 07:15:43 proc_lsst.multi:170 [INFO] [multi] job_id 16912359 proc_lsst.multi INFO: [multi] job_id 16912359 2024-03-08 07:15:43 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 07:15:43 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:43 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16912359 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16912359 parsl.executors.status_handling DEBUG: Launched block 75 on executor multi with job ID 16912359 parsl.executors.status_handling INFO: Allocated block ID 76 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-76 2024-03-08 07:15:43 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-76 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-76 2024-03-08 07:15:43 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:43 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:43 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:43 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 07:15:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:43 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 07:15:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:43 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 118 try 0 launched on executor multi with executor id 399 parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/f4ba88e8-9ec1-4d37-a2e0-cdd18c3b7056_characterizeImage_855042_2.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/f4ba88e8-9ec1-4d37-a2e0-cdd18c3b7056_characterizeImage_855042_2.stderr parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/15c7ee98-f29e-45b4-9520-12c93565eb34_isr_855042_2.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/15c7ee98-f29e-45b4-9520-12c93565eb34_isr_855042_2.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 9 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7467624ca3ca0e500066db9a25342c3e with result from task 9 parsl.dataflow.dflow DEBUG: Task 9 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 10 has memoization hash c050320f6182e34ae03dceca4bac4a4e parsl.dataflow.memoization INFO: Task 10 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467b2c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 10 try 0 launched on executor multi with executor id 400 parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/8841f386-834a-4d30-8f9b-386cdf36120b_characterizeImage_855137_61.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/8841f386-834a-4d30-8f9b-386cdf36120b_characterizeImage_855137_61.stderr parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/bfa98fd0-16d3-4f1c-9a2a-9c4ec194570e_isr_855137_61.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/bfa98fd0-16d3-4f1c-9a2a-9c4ec194570e_isr_855137_61.stderr parsl.dataflow.dflow INFO: Task 72 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7649f3328229f2fc2c987d30439a8ef1 with result from task 72 parsl.dataflow.dflow DEBUG: Task 72 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 73 has memoization hash f725b75d689a37f3a1e7e86a3d0b1374 parsl.dataflow.memoization INFO: Task 73 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d3060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 73 try 0 launched on executor multi with executor id 401 parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/7d77561c-108a-4042-90f6-a7f2c8f923bb_characterizeImage_855136_13.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/7d77561c-108a-4042-90f6-a7f2c8f923bb_characterizeImage_855136_13.stderr parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/253b3c8a-1782-454c-bb6a-7435ec419e0c_isr_855136_13.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/253b3c8a-1782-454c-bb6a-7435ec419e0c_isr_855136_13.stderr parsl.dataflow.dflow INFO: Task 18 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 573e418b898a50d3ec9a3ebf0fd4fd91 with result from task 18 parsl.dataflow.dflow DEBUG: Task 18 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 19 has memoization hash a9db8a832c5466615d7c1b37af88a27e parsl.dataflow.memoization INFO: Task 19 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467b3740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 19 try 0 launched on executor multi with executor id 402 parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/f0c42f63-1720-43d6-975a-b877f9843538_characterizeImage_855044_3.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/f0c42f63-1720-43d6-975a-b877f9843538_characterizeImage_855044_3.stderr parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/7d053fce-e1a9-44c6-9454-61852b4ad9e7_isr_855044_3.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/7d053fce-e1a9-44c6-9454-61852b4ad9e7_isr_855044_3.stderr parsl.dataflow.dflow INFO: Task 21 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4776766eb4a70a59850d944c0c6fbeb5 with result from task 21 parsl.dataflow.dflow DEBUG: Task 21 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 22 has memoization hash 96f2f2e1e0d4b57f9ebcaed8a43faaa5 parsl.dataflow.memoization INFO: Task 22 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467b3f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 22 try 0 launched on executor multi with executor id 403 parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/6915b873-8256-46f0-8d36-a7f567d3536d_characterizeImage_855137_52.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/6915b873-8256-46f0-8d36-a7f567d3536d_characterizeImage_855137_52.stderr parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/0b889644-816e-46a9-b3a2-90fa6534baf3_isr_855137_52.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/0b889644-816e-46a9-b3a2-90fa6534baf3_isr_855137_52.stderr 2024-03-08 07:15:43 proc_lsst.multi:170 [INFO] [multi] job_id 16912360 proc_lsst.multi INFO: [multi] job_id 16912360 2024-03-08 07:15:43 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-08 07:15:43 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:43 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16912360 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16912360 parsl.executors.status_handling DEBUG: Launched block 76 on executor multi with job ID 16912360 parsl.executors.status_handling INFO: Allocated block ID 77 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-77 2024-03-08 07:15:43 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-77 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-77 2024-03-08 07:15:43 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:43 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:43 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:43 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 07:15:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:43 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-08 07:15:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:43 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 111 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 08cca887a3a46fe4d5ff40808d734f71 with result from task 111 parsl.dataflow.dflow DEBUG: Task 111 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 112 has memoization hash 420ae5c45705c55896b50a68ae15d0ca parsl.dataflow.memoization INFO: Task 112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f5940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 112 try 0 launched on executor multi with executor id 404 parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/2f786d01-1cb7-4db6-a620-94a55caa1e6f_characterizeImage_855138_21.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/2f786d01-1cb7-4db6-a620-94a55caa1e6f_characterizeImage_855138_21.stderr parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/1b77263b-0d70-4b1a-b366-2c7005680ffb_isr_855138_21.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/1b77263b-0d70-4b1a-b366-2c7005680ffb_isr_855138_21.stderr parsl.dataflow.dflow INFO: Task 39 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46144cfd084f63f30b44cb0070e17542 with result from task 39 parsl.dataflow.dflow DEBUG: Task 39 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 40 has memoization hash c5cd2d395835751e630b18c6d4457f80 parsl.dataflow.memoization INFO: Task 40 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d0d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 40 try 0 launched on executor multi with executor id 405 parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/d948cbcf-5a02-4ab5-9cf1-c3a838990325_characterizeImage_855042_62.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/d948cbcf-5a02-4ab5-9cf1-c3a838990325_characterizeImage_855042_62.stderr parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/437c6816-f56c-4593-8bd0-0911f6628b9e_isr_855042_62.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/437c6816-f56c-4593-8bd0-0911f6628b9e_isr_855042_62.stderr parsl.dataflow.dflow INFO: Task 123 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab2b5104421e4c0c9031527e628cb842 with result from task 123 parsl.dataflow.dflow DEBUG: Task 123 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 124 has memoization hash 9302988dbe05646ef3e4ba6d6f251b65 parsl.dataflow.memoization INFO: Task 124 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f65c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 124 try 0 launched on executor multi with executor id 406 parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/9aa30944-e30a-4061-b9f3-a5d138b64ece_characterizeImage_855137_51.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/9aa30944-e30a-4061-b9f3-a5d138b64ece_characterizeImage_855137_51.stderr parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/91014b50-bbdf-441b-901b-e93dd03237e9_isr_855137_51.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/91014b50-bbdf-441b-901b-e93dd03237e9_isr_855137_51.stderr 2024-03-08 07:15:44 proc_lsst.multi:170 [INFO] [multi] job_id 16912361 proc_lsst.multi INFO: [multi] job_id 16912361 2024-03-08 07:15:44 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-08 07:15:44 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:44 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16912361 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16912361 parsl.executors.status_handling DEBUG: Launched block 77 on executor multi with job ID 16912361 parsl.executors.status_handling INFO: Allocated block ID 78 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-78 2024-03-08 07:15:44 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-78 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-78 2024-03-08 07:15:44 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:44 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:44 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:44 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 07:15:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:44 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-08 07:15:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:44 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 60 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98560f216fc5432aa4b120676fadd508 with result from task 60 parsl.dataflow.dflow DEBUG: Task 60 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 61 has memoization hash d9d7671408fb47acc568d597ae0de5a9 parsl.dataflow.memoization INFO: Task 61 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d23e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 61 try 0 launched on executor multi with executor id 407 parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/e184f266-fea7-41ec-995b-34cc43c5c41a_characterizeImage_855138_60.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/e184f266-fea7-41ec-995b-34cc43c5c41a_characterizeImage_855138_60.stderr parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/540a1e13-e110-4fdf-a686-428c42d17156_isr_855138_60.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/540a1e13-e110-4fdf-a686-428c42d17156_isr_855138_60.stderr parsl.dataflow.dflow INFO: Task 108 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33aa7c855b2184ce111e0bc6d9f964d2 with result from task 108 parsl.dataflow.dflow DEBUG: Task 108 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 109 has memoization hash 23b41625ce0c53d47a16761cd53e864d parsl.dataflow.memoization INFO: Task 109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f5620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 109 try 0 launched on executor multi with executor id 408 parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/327cb6b1-44ba-440f-a947-e7929b0e2e1c_characterizeImage_855044_21.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/327cb6b1-44ba-440f-a947-e7929b0e2e1c_characterizeImage_855044_21.stderr parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/2609987e-2bba-45ff-8d80-2f785eb100cd_isr_855044_21.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/2609987e-2bba-45ff-8d80-2f785eb100cd_isr_855044_21.stderr parsl.dataflow.dflow INFO: Task 99 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4bd380644eebb54ff71efff0b723e9f4 with result from task 99 parsl.dataflow.dflow DEBUG: Task 99 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 100 has memoization hash 5e6927f3529c5c127f71f6a393f5cd77 parsl.dataflow.memoization INFO: Task 100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f4cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 100 try 0 launched on executor multi with executor id 409 parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/ccc10e65-bd90-4813-ba42-c128f8516bbd_characterizeImage_855042_49.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/ccc10e65-bd90-4813-ba42-c128f8516bbd_characterizeImage_855042_49.stderr parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/660d9d5f-a50d-4fb7-b6d1-e15e64f6eae8_isr_855042_49.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/660d9d5f-a50d-4fb7-b6d1-e15e64f6eae8_isr_855042_49.stderr parsl.dataflow.dflow INFO: Task 120 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cdbd75bed5f97aee9e12698badb72262 with result from task 120 parsl.dataflow.dflow DEBUG: Task 120 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 121 has memoization hash 91b6558a4f587b9c8f55b9e34035b217 parsl.dataflow.memoization INFO: Task 121 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f62a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-08 07:15:44 proc_lsst.multi:170 [INFO] [multi] job_id 16912362 proc_lsst.multi INFO: [multi] job_id 16912362 2024-03-08 07:15:44 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-08 07:15:44 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:44 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16912362 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16912362 parsl.executors.status_handling DEBUG: Launched block 78 on executor multi with job ID 16912362 parsl.executors.status_handling INFO: Allocated block ID 79 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-79 2024-03-08 07:15:44 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-79 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-79 2024-03-08 07:15:44 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:44 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:44 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:44 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 07:15:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:44 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-08 07:15:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:44 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 121 try 0 launched on executor multi with executor id 410 parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/d61962d4-bdb2-4fc3-8552-48621f83cd07_characterizeImage_855044_57.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/d61962d4-bdb2-4fc3-8552-48621f83cd07_characterizeImage_855044_57.stderr parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/e65e2958-e469-4f77-a8cb-378dd89514e9_isr_855044_57.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/e65e2958-e469-4f77-a8cb-378dd89514e9_isr_855044_57.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 30 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 08f98d925f36d8bf810e82d63fcd9c04 with result from task 30 parsl.dataflow.dflow DEBUG: Task 30 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 31 has memoization hash 1c8779edab48d420ae9e5fe4b10be53a parsl.dataflow.memoization INFO: Task 31 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d0360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 31 try 0 launched on executor multi with executor id 411 parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/ce323cc8-de0d-4390-996f-f4fe562d5971_characterizeImage_855043_61.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/ce323cc8-de0d-4390-996f-f4fe562d5971_characterizeImage_855043_61.stderr parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/8c500fdd-f17c-4e5f-818d-307f8e868d29_isr_855043_61.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/8c500fdd-f17c-4e5f-818d-307f8e868d29_isr_855043_61.stderr parsl.dataflow.dflow INFO: Task 126 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f9b01e1129df6a45c64686aa1ba0086 with result from task 126 parsl.dataflow.dflow DEBUG: Task 126 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 127 has memoization hash ce5ca66c91dc6fc1decfb230c443e68a parsl.dataflow.memoization INFO: Task 127 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f68e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 127 try 0 launched on executor multi with executor id 412 parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/ff07f600-431e-42df-bc97-33ba2d86a5f1_characterizeImage_855136_42.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/ff07f600-431e-42df-bc97-33ba2d86a5f1_characterizeImage_855136_42.stderr parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/6eec908b-5279-4108-b93c-75d25440488e_isr_855136_42.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/6eec908b-5279-4108-b93c-75d25440488e_isr_855136_42.stderr 2024-03-08 07:15:45 proc_lsst.multi:170 [INFO] [multi] job_id 16912363 proc_lsst.multi INFO: [multi] job_id 16912363 2024-03-08 07:15:45 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-08 07:15:45 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:45 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16912363 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16912363 parsl.executors.status_handling DEBUG: Launched block 79 on executor multi with job ID 16912363 parsl.executors.status_handling INFO: Allocated block ID 80 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-80 2024-03-08 07:15:45 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-80 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-80 2024-03-08 07:15:45 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:45 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:45 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:45 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 07:15:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:45 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-08 07:15:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:45 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 159 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7dae35413fd80a74962643f5376fe88 with result from task 159 parsl.dataflow.dflow DEBUG: Task 159 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 160 has memoization hash 4cbf92743d38537927537e928bebe9c4 parsl.dataflow.memoization INFO: Task 160 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46814b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 160 try 0 launched on executor multi with executor id 413 parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/4010c5aa-b05f-42bf-8abc-a7bf4e73a658_characterizeImage_855138_57.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/4010c5aa-b05f-42bf-8abc-a7bf4e73a658_characterizeImage_855138_57.stderr parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/0d24145e-1ed2-42b0-880a-405e6601838b_isr_855138_57.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/0d24145e-1ed2-42b0-880a-405e6601838b_isr_855138_57.stderr 2024-03-08 07:15:46 proc_lsst.multi:170 [INFO] [multi] job_id 16912364 proc_lsst.multi INFO: [multi] job_id 16912364 2024-03-08 07:15:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-08 07:15:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:46 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16912364 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16912364 parsl.executors.status_handling DEBUG: Launched block 80 on executor multi with job ID 16912364 parsl.executors.status_handling INFO: Allocated block ID 81 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-81 2024-03-08 07:15:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-81 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-81 2024-03-08 07:15:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:46 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:46 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 07:15:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:46 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-08 07:15:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 75 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9b73ac692db53cb41d2ad2802eca884 with result from task 75 parsl.dataflow.dflow DEBUG: Task 75 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 76 has memoization hash a5f58bb156b08970cbe5fc644bb6b336 parsl.dataflow.memoization INFO: Task 76 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d3380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 76 try 0 launched on executor multi with executor id 414 parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/1323f941-885e-4fd3-8d78-5f903848d918_characterizeImage_855136_44.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/1323f941-885e-4fd3-8d78-5f903848d918_characterizeImage_855136_44.stderr parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/fcf92283-087a-42f0-bd66-fdc7f7f160be_isr_855136_44.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/fcf92283-087a-42f0-bd66-fdc7f7f160be_isr_855136_44.stderr parsl.dataflow.dflow INFO: Task 114 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7daef65b6711613e9c977e26918aa6a with result from task 114 parsl.dataflow.dflow DEBUG: Task 114 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 115 has memoization hash 6024ea3c4233b610985126d1a156686c parsl.dataflow.memoization INFO: Task 115 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f5c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 115 try 0 launched on executor multi with executor id 415 parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/a3a78a31-5494-49f9-84e7-1ce39cdd32e4_characterizeImage_855136_37.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/a3a78a31-5494-49f9-84e7-1ce39cdd32e4_characterizeImage_855136_37.stderr parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/e4432f35-43f8-4b63-a2f4-869159ee8586_isr_855136_37.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/e4432f35-43f8-4b63-a2f4-869159ee8586_isr_855136_37.stderr parsl.dataflow.dflow INFO: Task 129 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0bd2b1b23d45065e743d29bcb1130163 with result from task 129 parsl.dataflow.dflow DEBUG: Task 129 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 130 has memoization hash 4bc04a5cd170a517d8ba58deeea89adc parsl.dataflow.memoization INFO: Task 130 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f6c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 130 try 0 launched on executor multi with executor id 416 parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/44388d39-c1f5-4643-8410-46500f9c9f82_characterizeImage_855042_3.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/44388d39-c1f5-4643-8410-46500f9c9f82_characterizeImage_855042_3.stderr parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/39c656c9-b606-45cb-8172-24f3cd5f4289_isr_855042_3.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/39c656c9-b606-45cb-8172-24f3cd5f4289_isr_855042_3.stderr 2024-03-08 07:15:46 proc_lsst.multi:170 [INFO] [multi] job_id 16912365 proc_lsst.multi INFO: [multi] job_id 16912365 2024-03-08 07:15:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-08 07:15:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:46 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16912365 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16912365 parsl.executors.status_handling DEBUG: Launched block 81 on executor multi with job ID 16912365 parsl.executors.status_handling INFO: Allocated block ID 82 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-82 2024-03-08 07:15:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-82 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-82 2024-03-08 07:15:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:46 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:46 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 07:15:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:46 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-08 07:15:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 78 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fab42e62c82b9091ecc1a4defcb15d8f with result from task 78 parsl.dataflow.dflow DEBUG: Task 78 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 79 has memoization hash 8e752a7b4cb8c92afeb74287cc1fdc28 parsl.dataflow.memoization INFO: Task 79 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d36a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 79 try 0 launched on executor multi with executor id 417 parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/7c458fee-c571-466e-b4c7-e9f47152600e_characterizeImage_855136_60.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/7c458fee-c571-466e-b4c7-e9f47152600e_characterizeImage_855136_60.stderr parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/17e53291-19c2-424a-83b6-f5d90227be32_isr_855136_60.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/17e53291-19c2-424a-83b6-f5d90227be32_isr_855136_60.stderr 2024-03-08 07:15:46 proc_lsst.multi:170 [INFO] [multi] job_id 16912366 proc_lsst.multi INFO: [multi] job_id 16912366 2024-03-08 07:15:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-08 07:15:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:46 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16912366 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16912366 parsl.executors.status_handling DEBUG: Launched block 82 on executor multi with job ID 16912366 parsl.executors.status_handling INFO: Allocated block ID 83 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-83 2024-03-08 07:15:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-83 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-83 2024-03-08 07:15:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:46 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:46 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 07:15:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:46 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-08 07:15:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:47 proc_lsst.multi:170 [INFO] [multi] job_id 16912367 proc_lsst.multi INFO: [multi] job_id 16912367 2024-03-08 07:15:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-08 07:15:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:47 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16912367 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16912367 parsl.executors.status_handling DEBUG: Launched block 83 on executor multi with job ID 16912367 parsl.executors.status_handling INFO: Allocated block ID 84 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-84 2024-03-08 07:15:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-84 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-84 2024-03-08 07:15:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:47 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:47 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 07:15:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:47 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-08 07:15:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:47 proc_lsst.multi:170 [INFO] [multi] job_id 16912368 proc_lsst.multi INFO: [multi] job_id 16912368 2024-03-08 07:15:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-08 07:15:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:47 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16912368 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16912368 parsl.executors.status_handling DEBUG: Launched block 84 on executor multi with job ID 16912368 parsl.executors.status_handling INFO: Allocated block ID 85 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-85 2024-03-08 07:15:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-85 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-85 2024-03-08 07:15:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:47 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:47 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 07:15:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:47 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-08 07:15:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:48 proc_lsst.multi:170 [INFO] [multi] job_id 16912369 proc_lsst.multi INFO: [multi] job_id 16912369 2024-03-08 07:15:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:48 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16912369 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16912369 parsl.executors.status_handling DEBUG: Launched block 85 on executor multi with job ID 16912369 parsl.executors.status_handling INFO: Allocated block ID 86 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-86 2024-03-08 07:15:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-86 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-86 2024-03-08 07:15:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:48 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:48 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 07:15:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:48 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:48 proc_lsst.multi:170 [INFO] [multi] job_id 16912370 proc_lsst.multi INFO: [multi] job_id 16912370 2024-03-08 07:15:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 31 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 31 2024-03-08 07:15:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:48 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16912370 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16912370 parsl.executors.status_handling DEBUG: Launched block 86 on executor multi with job ID 16912370 parsl.executors.status_handling INFO: Allocated block ID 87 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-87 2024-03-08 07:15:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-87 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-87 2024-03-08 07:15:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:48 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:48 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 07:15:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:48 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 31 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 31 2024-03-08 07:15:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:49 proc_lsst.multi:170 [INFO] [multi] job_id 16912371 proc_lsst.multi INFO: [multi] job_id 16912371 2024-03-08 07:15:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 32 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 32 2024-03-08 07:15:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:49 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16912371 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16912371 parsl.executors.status_handling DEBUG: Launched block 87 on executor multi with job ID 16912371 parsl.executors.status_handling INFO: Allocated block ID 88 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-88 2024-03-08 07:15:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-88 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-88 2024-03-08 07:15:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:49 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:49 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 07:15:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:49 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 32 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 32 2024-03-08 07:15:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:49 proc_lsst.multi:170 [INFO] [multi] job_id 16912372 proc_lsst.multi INFO: [multi] job_id 16912372 2024-03-08 07:15:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 33 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 33 2024-03-08 07:15:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:49 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16912372 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16912372 parsl.executors.status_handling DEBUG: Launched block 88 on executor multi with job ID 16912372 parsl.executors.status_handling INFO: Allocated block ID 89 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-89 2024-03-08 07:15:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-89 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-89 2024-03-08 07:15:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:49 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:49 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 07:15:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:49 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 33 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 33 2024-03-08 07:15:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:50 proc_lsst.multi:170 [INFO] [multi] job_id 16912373 proc_lsst.multi INFO: [multi] job_id 16912373 2024-03-08 07:15:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 34 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 34 2024-03-08 07:15:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:50 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16912373 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16912373 parsl.executors.status_handling DEBUG: Launched block 89 on executor multi with job ID 16912373 parsl.executors.status_handling INFO: Allocated block ID 90 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-90 2024-03-08 07:15:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-90 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-90 2024-03-08 07:15:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:50 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:50 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 07:15:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:50 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 34 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 34 2024-03-08 07:15:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:15:51 proc_lsst.multi:170 [INFO] [multi] job_id 16912374 proc_lsst.multi INFO: [multi] job_id 16912374 2024-03-08 07:15:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 35 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 35 2024-03-08 07:15:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:51 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16912374 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16912374 parsl.executors.status_handling DEBUG: Launched block 90 on executor multi with job ID 16912374 parsl.executors.status_handling INFO: Allocated block ID 91 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-91 2024-03-08 07:15:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-91 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-91 2024-03-08 07:15:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:15:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:15:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:15:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:15:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:15:51 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:15:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:15:51 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 07:15:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:15:51 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:15:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 35 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 35 2024-03-08 07:15:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:15:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 135 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ebcee17d6f90a2aa2c23ae3e4e321d7 with result from task 135 parsl.dataflow.dflow DEBUG: Task 135 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 136 has memoization hash 4359779f50d07b6bb30ddd1a64e8b851 parsl.dataflow.memoization INFO: Task 136 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f7240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 136 try 0 launched on executor multi with executor id 418 parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/41023e02-839f-41c9-9219-f53c0366a87b_characterizeImage_855137_57.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/41023e02-839f-41c9-9219-f53c0366a87b_characterizeImage_855137_57.stderr parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/2ca50aa6-1a26-4a54-b2f0-232bbcc7364c_isr_855137_57.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/2ca50aa6-1a26-4a54-b2f0-232bbcc7364c_isr_855137_57.stderr parsl.dataflow.dflow INFO: Task 132 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 897ad8657a6476ede461994d2cc40277 with result from task 132 parsl.dataflow.dflow DEBUG: Task 132 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 133 has memoization hash 14d4850e1684ac9ec708a1393c5b7ff1 parsl.dataflow.memoization INFO: Task 133 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f6f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 133 try 0 launched on executor multi with executor id 419 parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/12083d16-f0d7-4066-9e0c-c0e520449e58_characterizeImage_855044_45.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/12083d16-f0d7-4066-9e0c-c0e520449e58_characterizeImage_855044_45.stderr parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/cd2f75e7-2322-4a73-af55-fd6ad666dd2a_isr_855044_45.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/cd2f75e7-2322-4a73-af55-fd6ad666dd2a_isr_855044_45.stderr parsl.dataflow.dflow INFO: Task 138 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02ac701d3db3800fe8d9ac011a6570b7 with result from task 138 parsl.dataflow.dflow DEBUG: Task 138 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 139 has memoization hash d316f8d6bab4ae779e2f370de47c9277 parsl.dataflow.memoization INFO: Task 139 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f7560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 139 try 0 launched on executor multi with executor id 420 parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/69f0e87f-a264-4efb-9dbe-c7dd4f51af85_characterizeImage_855044_34.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/69f0e87f-a264-4efb-9dbe-c7dd4f51af85_characterizeImage_855044_34.stderr parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/f0fbb9f6-d361-495b-95b6-2f50813275e4_isr_855044_34.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/f0fbb9f6-d361-495b-95b6-2f50813275e4_isr_855044_34.stderr parsl.dataflow.dflow INFO: Task 141 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c17fb59da70de0c91f96a8b135084d2 with result from task 141 parsl.dataflow.dflow DEBUG: Task 141 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 142 has memoization hash a82a8bd1b7c34ddbdecd6cca772d41e6 parsl.dataflow.memoization INFO: Task 142 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f7880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 142 try 0 launched on executor multi with executor id 421 parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/e9b5c366-bda1-4b4a-9a99-f90ee7af7a3f_characterizeImage_855044_24.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/e9b5c366-bda1-4b4a-9a99-f90ee7af7a3f_characterizeImage_855044_24.stderr parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/88e3bc28-9b06-4833-b52d-802f02ca873d_isr_855044_24.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/88e3bc28-9b06-4833-b52d-802f02ca873d_isr_855044_24.stderr parsl.dataflow.dflow INFO: Task 150 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ad971c09e4c47be235d056266b249f6 with result from task 150 parsl.dataflow.dflow DEBUG: Task 150 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 151 has memoization hash fb67acbd45604b457b27824259626780 parsl.dataflow.memoization INFO: Task 151 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46814220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 151 try 0 launched on executor multi with executor id 422 parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/65fd74c5-4005-4161-9163-db77bd4a52f4_characterizeImage_855043_29.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/65fd74c5-4005-4161-9163-db77bd4a52f4_characterizeImage_855043_29.stderr parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/d4729193-6884-40e6-bf93-7334472f419e_isr_855043_29.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/d4729193-6884-40e6-bf93-7334472f419e_isr_855043_29.stderr parsl.dataflow.dflow INFO: Task 147 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4120ada7ea71a5f7a806145ce720dabb with result from task 147 parsl.dataflow.dflow DEBUG: Task 147 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 148 has memoization hash 7bac5eb8c93568841d05d9ccb8b921db parsl.dataflow.memoization INFO: Task 148 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f7ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 148 try 0 launched on executor multi with executor id 423 parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/ee82dcd2-77a5-4432-8f0e-6f9dfbef9e75_characterizeImage_855138_37.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/ee82dcd2-77a5-4432-8f0e-6f9dfbef9e75_characterizeImage_855138_37.stderr parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/b82f5527-1e2e-4403-a810-9fb23276c8bd_isr_855138_37.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/b82f5527-1e2e-4403-a810-9fb23276c8bd_isr_855138_37.stderr parsl.dataflow.dflow INFO: Task 144 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d5bcc7b86757a903ac35a91f786ade6 with result from task 144 parsl.dataflow.dflow DEBUG: Task 144 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 145 has memoization hash 0569beb8e6be32cb8ef4189757ed372a parsl.dataflow.memoization INFO: Task 145 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f7ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 145 try 0 launched on executor multi with executor id 424 parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/f8887070-aa46-408d-9b36-e3a52a4f5430_characterizeImage_855042_45.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/f8887070-aa46-408d-9b36-e3a52a4f5430_characterizeImage_855042_45.stderr parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/c8ca8d8e-01e3-48c2-9e5e-2eeff2e48ab6_isr_855042_45.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/c8ca8d8e-01e3-48c2-9e5e-2eeff2e48ab6_isr_855042_45.stderr parsl.dataflow.dflow INFO: Task 183 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca49b363bcee2def3c4814c7f32c8cfa with result from task 183 parsl.dataflow.dflow DEBUG: Task 183 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 184 has memoization hash 738c5b454ac0816ea11364c9ceeeeae2 parsl.dataflow.memoization INFO: Task 184 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46816480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 184 try 0 launched on executor multi with executor id 425 parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/edeadeec-41b9-44c8-9832-6740d2ce423e_characterizeImage_855137_6.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/edeadeec-41b9-44c8-9832-6740d2ce423e_characterizeImage_855137_6.stderr parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/6c5bf872-099f-40ce-a235-0b23b31e0abd_isr_855137_6.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/6c5bf872-099f-40ce-a235-0b23b31e0abd_isr_855137_6.stderr parsl.dataflow.dflow INFO: Task 165 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6bbb0abf54ab7558b8100bafa9fcf921 with result from task 165 parsl.dataflow.dflow DEBUG: Task 165 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 166 has memoization hash 73e28589856c2e71bf4ca91ce258a940 parsl.dataflow.memoization INFO: Task 166 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa468151c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 166 try 0 launched on executor multi with executor id 426 parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/1def5661-9106-45ed-8663-96a3d09891d5_characterizeImage_855042_20.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/1def5661-9106-45ed-8663-96a3d09891d5_characterizeImage_855042_20.stderr parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/898687d0-0e30-4ee7-b8a9-f5cab23999b5_isr_855042_20.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/898687d0-0e30-4ee7-b8a9-f5cab23999b5_isr_855042_20.stderr parsl.dataflow.dflow INFO: Task 162 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd4d7ac0138224c419ac4863a4b16224 with result from task 162 parsl.dataflow.dflow DEBUG: Task 162 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 163 has memoization hash ff9690ffe4be9be9d52fdf1209e90270 parsl.dataflow.memoization INFO: Task 163 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46814ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 163 try 0 launched on executor multi with executor id 427 parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/44527b48-1ff5-4e77-aa8c-b87d9b549060_characterizeImage_855042_35.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/44527b48-1ff5-4e77-aa8c-b87d9b549060_characterizeImage_855042_35.stderr parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/dddd104b-3799-4bb2-8d3b-6118a5f03349_isr_855042_35.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/dddd104b-3799-4bb2-8d3b-6118a5f03349_isr_855042_35.stderr parsl.dataflow.dflow INFO: Task 153 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3cb700137e4d49b8cd17a89cabd1e9a3 with result from task 153 parsl.dataflow.dflow DEBUG: Task 153 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 154 has memoization hash 5a5d63a71a908e2e10a1135bcf28dd83 parsl.dataflow.memoization INFO: Task 154 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa468142c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 154 try 0 launched on executor multi with executor id 428 parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/6ce5fb69-6137-49e5-b89d-5e80edcd4762_characterizeImage_855138_56.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/6ce5fb69-6137-49e5-b89d-5e80edcd4762_characterizeImage_855138_56.stderr parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/7ee269cb-5ebb-4149-b13a-7f9461be3e9e_isr_855138_56.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/7ee269cb-5ebb-4149-b13a-7f9461be3e9e_isr_855138_56.stderr parsl.dataflow.dflow INFO: Task 210 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e70ec555dd4406c4854ab4611e51831f with result from task 210 parsl.dataflow.dflow DEBUG: Task 210 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 211 has memoization hash 93fa0248c7c7cdcb8790de11515c59a3 parsl.dataflow.memoization INFO: Task 211 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46817ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 211 try 0 launched on executor multi with executor id 429 parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/6cc4cb45-3d86-4472-b71a-85186ee9aedb_characterizeImage_855137_33.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/6cc4cb45-3d86-4472-b71a-85186ee9aedb_characterizeImage_855137_33.stderr parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/a117a6fa-232a-4b19-8fce-18f3cef48883_isr_855137_33.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/a117a6fa-232a-4b19-8fce-18f3cef48883_isr_855137_33.stderr parsl.dataflow.dflow INFO: Task 186 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad775c156c6bad418000312f3adc78fb with result from task 186 parsl.dataflow.dflow DEBUG: Task 186 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 187 has memoization hash cbdf5cc0d0b23777c724475dd911ca94 parsl.dataflow.memoization INFO: Task 187 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa468167a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 187 try 0 launched on executor multi with executor id 430 parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/92b997cf-ec4a-4a21-8f3d-aed010c3f96b_characterizeImage_855044_44.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/92b997cf-ec4a-4a21-8f3d-aed010c3f96b_characterizeImage_855044_44.stderr parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/5b1028ef-b906-4f28-a08a-4d4859621ffb_isr_855044_44.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/5b1028ef-b906-4f28-a08a-4d4859621ffb_isr_855044_44.stderr parsl.dataflow.dflow INFO: Task 156 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8197dafb71467eb7bfbe9c0fb0f75da8 with result from task 156 parsl.dataflow.dflow DEBUG: Task 156 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 157 has memoization hash ed9c59e73e3d9a3b855e260b52b2186f parsl.dataflow.memoization INFO: Task 157 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46814860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 157 try 0 launched on executor multi with executor id 431 parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/ab985c4a-d0a3-42a2-b0f0-91438c6d5169_characterizeImage_855043_55.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/ab985c4a-d0a3-42a2-b0f0-91438c6d5169_characterizeImage_855043_55.stderr parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/65becc69-abba-4e78-9acf-e8166c3c152d_isr_855043_55.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/65becc69-abba-4e78-9acf-e8166c3c152d_isr_855043_55.stderr parsl.dataflow.dflow INFO: Task 180 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72697841cfc64beb4af3d64be7bac29b with result from task 180 parsl.dataflow.dflow DEBUG: Task 180 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 181 has memoization hash a2e797a59871f6d34372f3b3745a98fc parsl.dataflow.memoization INFO: Task 181 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46816160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 181 try 0 launched on executor multi with executor id 432 parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/66943aca-617f-482c-be6a-bda81775548a_characterizeImage_855044_58.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/66943aca-617f-482c-be6a-bda81775548a_characterizeImage_855044_58.stderr parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/4ff7ad7f-c353-4ec9-a506-c0cb052d32eb_isr_855044_58.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/4ff7ad7f-c353-4ec9-a506-c0cb052d32eb_isr_855044_58.stderr parsl.dataflow.dflow INFO: Task 177 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5733044278bc75a02b99e2d91e9d8f8a with result from task 177 parsl.dataflow.dflow DEBUG: Task 177 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 178 has memoization hash c4e59f2692aacbc2679ea08240abb5ea parsl.dataflow.memoization INFO: Task 178 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46815e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 178 try 0 launched on executor multi with executor id 433 parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/1e99a0ed-f7c4-4fdc-a136-60284dcc1c0a_characterizeImage_855137_62.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/1e99a0ed-f7c4-4fdc-a136-60284dcc1c0a_characterizeImage_855137_62.stderr parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/3800fef7-f63b-4327-98e3-3ba1cc2637d3_isr_855137_62.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/3800fef7-f63b-4327-98e3-3ba1cc2637d3_isr_855137_62.stderr parsl.dataflow.dflow INFO: Task 189 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8171df3ebe51bff72a33b32724dce09 with result from task 189 parsl.dataflow.dflow DEBUG: Task 189 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 190 has memoization hash 2c30e9ba708ae6eb2c52972eaff08763 parsl.dataflow.memoization INFO: Task 190 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46816ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 190 try 0 launched on executor multi with executor id 434 parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/2a2c6661-715b-4b61-9883-5e6a9da92bd3_characterizeImage_855044_60.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/2a2c6661-715b-4b61-9883-5e6a9da92bd3_characterizeImage_855044_60.stderr parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/36c871f6-0028-4c71-8df5-daa791642dbd_isr_855044_60.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/36c871f6-0028-4c71-8df5-daa791642dbd_isr_855044_60.stderr parsl.dataflow.dflow INFO: Task 207 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dde581b5fe85abb1ac32305bb8a85fa7 with result from task 207 parsl.dataflow.dflow DEBUG: Task 207 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 208 has memoization hash 5ecad099e0db6b7cd46973c9a6ee4424 parsl.dataflow.memoization INFO: Task 208 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46817d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 208 try 0 launched on executor multi with executor id 435 parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/fb0a4c66-200e-4adf-a6b8-30eb7e648b09_characterizeImage_855042_17.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/fb0a4c66-200e-4adf-a6b8-30eb7e648b09_characterizeImage_855042_17.stderr parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/19394193-1a40-4edf-bc70-5a882a580a65_isr_855042_17.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/19394193-1a40-4edf-bc70-5a882a580a65_isr_855042_17.stderr parsl.dataflow.dflow INFO: Task 228 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 88ca0a9a10653fa6d8a14c1fc6241ba4 with result from task 228 parsl.dataflow.dflow DEBUG: Task 228 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 229 has memoization hash f3e917f7aa5058ae823b69bc6de1101c parsl.dataflow.memoization INFO: Task 229 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa468353a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 229 try 0 launched on executor multi with executor id 436 parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/dff41e58-7500-46f3-b543-95f755274de4_characterizeImage_855137_32.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/dff41e58-7500-46f3-b543-95f755274de4_characterizeImage_855137_32.stderr parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/9fc95394-35ff-49aa-84ba-359555b5041c_isr_855137_32.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/9fc95394-35ff-49aa-84ba-359555b5041c_isr_855137_32.stderr parsl.dataflow.dflow INFO: Task 168 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 126fe71ff91b0402f19333144fe46f28 with result from task 168 parsl.dataflow.dflow DEBUG: Task 168 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 169 has memoization hash 9158f6f3a28498edde514bd5f4316e5b parsl.dataflow.memoization INFO: Task 169 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa468154e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 169 try 0 launched on executor multi with executor id 437 parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/f478c8e6-13d6-4368-8938-660b253e16d3_characterizeImage_855042_39.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/f478c8e6-13d6-4368-8938-660b253e16d3_characterizeImage_855042_39.stderr parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/3823d9b7-ef0e-4a4e-89cf-aad04a6be695_isr_855042_39.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/3823d9b7-ef0e-4a4e-89cf-aad04a6be695_isr_855042_39.stderr parsl.dataflow.dflow INFO: Task 171 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0361ab0ed6fe33abe77aac5b2140dbe2 with result from task 171 parsl.dataflow.dflow DEBUG: Task 171 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 172 has memoization hash e90c6592c3c15e1ab6eb217f5aa1fd06 parsl.dataflow.memoization INFO: Task 172 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46815800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 172 try 0 launched on executor multi with executor id 438 parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/96c60d0d-3417-4a6c-9754-6a5095204de4_characterizeImage_855138_49.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/96c60d0d-3417-4a6c-9754-6a5095204de4_characterizeImage_855138_49.stderr parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/ffb275d3-f855-4378-93c8-22bb9eef0d15_isr_855138_49.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/ffb275d3-f855-4378-93c8-22bb9eef0d15_isr_855138_49.stderr parsl.dataflow.dflow INFO: Task 270 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 41e6f77a022972d54320c3669b8847ae with result from task 270 parsl.dataflow.dflow DEBUG: Task 270 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 271 has memoization hash 96b98422f8159adc65593fb1bb06a7d7 parsl.dataflow.memoization INFO: Task 271 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46837f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 271 try 0 launched on executor multi with executor id 439 parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/ef65dffc-6a37-4620-b178-4caa1c058ea3_characterizeImage_855042_16.stdout parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/ef65dffc-6a37-4620-b178-4caa1c058ea3_characterizeImage_855042_16.stderr parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/bd2aff70-d916-45a0-b649-f9c06405b49c_isr_855042_16.stdout parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/bd2aff70-d916-45a0-b649-f9c06405b49c_isr_855042_16.stderr parsl.dataflow.dflow INFO: Task 264 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24da08f7f2eb10fa17a82a08037b36f9 with result from task 264 parsl.dataflow.dflow DEBUG: Task 264 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 265 has memoization hash cafcf8df6ccc1b224928b89886eecd95 parsl.dataflow.memoization INFO: Task 265 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46837920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 265 try 0 launched on executor multi with executor id 440 parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/d113dfd3-beb2-40bb-b845-771b99a21647_characterizeImage_855044_40.stdout parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/d113dfd3-beb2-40bb-b845-771b99a21647_characterizeImage_855044_40.stderr parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/0e8f3fbe-0214-4d49-9a6a-9ac27a09f774_isr_855044_40.stdout parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/0e8f3fbe-0214-4d49-9a6a-9ac27a09f774_isr_855044_40.stderr parsl.dataflow.dflow INFO: Task 366 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6f972905be8aa134dd42e8fc6cc46124 with result from task 366 parsl.dataflow.dflow DEBUG: Task 366 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 367 has memoization hash 6dd50be2840f3a7051b2b03b582b0010 parsl.dataflow.memoization INFO: Task 367 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4687a3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 367 try 0 launched on executor multi with executor id 441 parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/d32e73d8-8360-4b1a-9b1d-888889c9057c_characterizeImage_855044_17.stdout parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/d32e73d8-8360-4b1a-9b1d-888889c9057c_characterizeImage_855044_17.stderr parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/97b5307d-64bd-4d9a-afc0-248a428a35a5_isr_855044_17.stdout parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/97b5307d-64bd-4d9a-afc0-248a428a35a5_isr_855044_17.stderr parsl.dataflow.dflow INFO: Task 288 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03b31bebf768d48ce8a2794197395af4 with result from task 288 parsl.dataflow.dflow DEBUG: Task 288 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 289 has memoization hash f6351b9b4d460a720eb496b0cbfbf077 parsl.dataflow.memoization INFO: Task 289 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 289 try 0 launched on executor multi with executor id 442 parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/9c5712a2-b7f4-4b15-a542-0d04640e0e4a_characterizeImage_855138_13.stdout parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/9c5712a2-b7f4-4b15-a542-0d04640e0e4a_characterizeImage_855138_13.stderr parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/c17cb89b-dff2-4716-9bcd-ca79783d2105_isr_855138_13.stdout parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/c17cb89b-dff2-4716-9bcd-ca79783d2105_isr_855138_13.stderr parsl.dataflow.dflow INFO: Task 291 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c4ef731e764f3589441608c7624b92ef with result from task 291 parsl.dataflow.dflow DEBUG: Task 291 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 292 has memoization hash 1f0a2a10aca1ec98a24f02e34c5e3ff7 parsl.dataflow.memoization INFO: Task 292 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 292 try 0 launched on executor multi with executor id 443 parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/9cd770ed-83d4-441d-b93c-c75c12ab3386_characterizeImage_855044_12.stdout parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/9cd770ed-83d4-441d-b93c-c75c12ab3386_characterizeImage_855044_12.stderr parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/19130f1e-5ef1-496a-91dc-6dcb75ff9e92_isr_855044_12.stdout parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/19130f1e-5ef1-496a-91dc-6dcb75ff9e92_isr_855044_12.stderr parsl.dataflow.dflow INFO: Task 282 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8833682497ceb7d0a9b154dba10eeb7f with result from task 282 parsl.dataflow.dflow DEBUG: Task 282 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 283 has memoization hash d8df2da1b1a572a7a4056ff897a99cda parsl.dataflow.memoization INFO: Task 283 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 283 try 0 launched on executor multi with executor id 444 parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/4541dd72-ab0b-4cad-9e69-75528dc83160_characterizeImage_855043_36.stdout parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/4541dd72-ab0b-4cad-9e69-75528dc83160_characterizeImage_855043_36.stderr parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/7a899baf-353e-4b0f-8e09-6872269e794b_isr_855043_36.stdout parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/7a899baf-353e-4b0f-8e09-6872269e794b_isr_855043_36.stderr parsl.dataflow.dflow INFO: Task 369 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df80d5d596f8322b2e26c80af3749de9 with result from task 369 parsl.dataflow.dflow DEBUG: Task 369 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 370 has memoization hash a7f421a68c2a2526a1512ebaecec823e parsl.dataflow.memoization INFO: Task 370 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4687a700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 370 try 0 launched on executor multi with executor id 445 parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/d105a0b1-b098-4508-8693-9806986cfac7_characterizeImage_855137_3.stdout parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/d105a0b1-b098-4508-8693-9806986cfac7_characterizeImage_855137_3.stderr parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/6d74f0ca-c255-4305-ad23-73539a9dea64_isr_855137_3.stdout parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/6d74f0ca-c255-4305-ad23-73539a9dea64_isr_855137_3.stderr parsl.dataflow.dflow INFO: Task 222 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dae6ce8fd412b65cb5151f1a76bed5eb with result from task 222 parsl.dataflow.dflow DEBUG: Task 222 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 223 has memoization hash 10753c3c50ecaac721e4da8d5680e07a parsl.dataflow.memoization INFO: Task 223 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46834d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 223 try 0 launched on executor multi with executor id 446 parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/63a5eb83-4b2f-4c56-9880-64ccddeef7a2_characterizeImage_855044_32.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/63a5eb83-4b2f-4c56-9880-64ccddeef7a2_characterizeImage_855044_32.stderr parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/9dde3eb1-1348-411c-8d60-34eff42df87e_isr_855044_32.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/9dde3eb1-1348-411c-8d60-34eff42df87e_isr_855044_32.stderr parsl.dataflow.dflow INFO: Task 276 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b60bbbb983792035843eb6c9d9667c5 with result from task 276 parsl.dataflow.dflow DEBUG: Task 276 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 277 has memoization hash 8681e04e2578377f621ab461b7458f4b parsl.dataflow.memoization INFO: Task 277 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 277 try 0 launched on executor multi with executor id 447 parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/f7352a57-b317-4e47-8790-85f807bfc37c_characterizeImage_855137_36.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/f7352a57-b317-4e47-8790-85f807bfc37c_characterizeImage_855137_36.stderr parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/4a27895d-7adf-49dc-9881-0311f13fb2e5_isr_855137_36.stdout parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/4a27895d-7adf-49dc-9881-0311f13fb2e5_isr_855137_36.stderr parsl.dataflow.dflow INFO: Task 240 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8f1807c6ea7cfeaeb89e751173c3320 with result from task 240 parsl.dataflow.dflow DEBUG: Task 240 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 241 has memoization hash 97b7a7c494fc5dcab8a98ec6fa8cd47e parsl.dataflow.memoization INFO: Task 241 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46836020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 241 try 0 launched on executor multi with executor id 448 parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/a461e955-6651-4a33-9764-d79444da84f4_characterizeImage_855136_8.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/a461e955-6651-4a33-9764-d79444da84f4_characterizeImage_855136_8.stderr parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/0ab5ff96-5836-43a9-a6b4-6f7e0b17945a_isr_855136_8.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/0ab5ff96-5836-43a9-a6b4-6f7e0b17945a_isr_855136_8.stderr parsl.channels.local.local WARNING: Execution of command 'sbatch /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/submit_scripts/parsl.multi.block-91.1709910951.01504.submit' failed due to Command 'sbatch /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/submit_scripts/parsl.multi.block-91.1709910951.01504.submit' timed out after 10 seconds parsl.executors.status_handling INFO: Allocated block ID 92 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-92 2024-03-08 07:16:01 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-92 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-92 2024-03-08 07:16:01 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:16:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:16:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:16:01 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:16:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:16:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:16:01 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:16:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:16:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:16:01 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:16:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 07:16:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:16:01 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:16:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 35 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 35 2024-03-08 07:16:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:16:01 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 243 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 151dea84198ad2a7eb38407dfe528f30 with result from task 243 parsl.dataflow.dflow DEBUG: Task 243 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 244 has memoization hash e376ac8f10530129fd53efd2a6401d55 parsl.dataflow.memoization INFO: Task 244 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46836340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 244 try 0 launched on executor multi with executor id 449 parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/cecac4d3-2ee8-4834-b131-4e489d30b935_characterizeImage_855043_13.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/cecac4d3-2ee8-4834-b131-4e489d30b935_characterizeImage_855043_13.stderr parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/1e777f90-1c49-4e07-b97a-6bc34ef90168_isr_855043_13.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/1e777f90-1c49-4e07-b97a-6bc34ef90168_isr_855043_13.stderr parsl.dataflow.dflow INFO: Task 219 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3bdd8e02cbbcdef044f0db3e0752ea1b with result from task 219 parsl.dataflow.dflow DEBUG: Task 219 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 220 has memoization hash 3b94649128ddc57d620c00367ceebbb9 parsl.dataflow.memoization INFO: Task 220 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46834a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 220 try 0 launched on executor multi with executor id 450 parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/10aa27db-ff53-4f11-9f19-9709837bafe7_characterizeImage_855043_32.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/10aa27db-ff53-4f11-9f19-9709837bafe7_characterizeImage_855043_32.stderr parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/b37a1860-7784-41e7-9b87-336efae3ea1d_isr_855043_32.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/b37a1860-7784-41e7-9b87-336efae3ea1d_isr_855043_32.stderr parsl.dataflow.dflow INFO: Task 192 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba6a1d805b868573074901cc4638f9b4 with result from task 192 parsl.dataflow.dflow DEBUG: Task 192 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 193 has memoization hash 26ac35739269d2ec230b77897efc0845 parsl.dataflow.memoization INFO: Task 193 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46816de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 193 try 0 launched on executor multi with executor id 451 parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/2b849466-3b1a-41a0-9353-96411c953838_characterizeImage_855136_33.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/2b849466-3b1a-41a0-9353-96411c953838_characterizeImage_855136_33.stderr parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/73856558-3994-454b-869e-10ae4e73aedf_isr_855136_33.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/73856558-3994-454b-869e-10ae4e73aedf_isr_855136_33.stderr parsl.dataflow.dflow INFO: Task 204 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c2fb1676b6c266a99dbd4a574af6d46 with result from task 204 parsl.dataflow.dflow DEBUG: Task 204 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 205 has memoization hash 0a43b8dc195376d9c5ede773b5acbbb2 parsl.dataflow.memoization INFO: Task 205 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46817a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 205 try 0 launched on executor multi with executor id 452 parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/e26f148d-c591-49ca-9a5c-5cf84a33b4c1_characterizeImage_855044_33.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/e26f148d-c591-49ca-9a5c-5cf84a33b4c1_characterizeImage_855044_33.stderr parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/862d6136-bf44-46a8-8c2e-364e100f2401_isr_855044_33.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/862d6136-bf44-46a8-8c2e-364e100f2401_isr_855044_33.stderr 2024-03-08 07:16:01 proc_lsst.multi:170 [INFO] [multi] job_id 16912375 proc_lsst.multi INFO: [multi] job_id 16912375 2024-03-08 07:16:01 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 36 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 36 2024-03-08 07:16:01 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:16:01 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16912375 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16912375 parsl.executors.status_handling DEBUG: Launched block 92 on executor multi with job ID 16912375 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 2024-03-08 07:16:01 proc_lsst.multi:146 [INFO] found job 75775 in provider local proc_lsst.multi INFO: found job 75775 in provider local parsl.dataflow.dflow INFO: Task 372 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6295370b6b200d25c462c0203cbb28c4 with result from task 372 2024-03-08 07:16:01 proc_lsst.multi:146 [INFO] found job 16912283 in provider astro proc_lsst.multi INFO: found job 16912283 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow DEBUG: Task 372 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 373 has memoization hash 9ef98ff2eb21a4736bf70f10ee3245e4 parsl.dataflow.memoization INFO: Task 373 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4687aa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 373 try 0 launched on executor multi with executor id 453 parsl.dataflow.dflow INFO: Standard output for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/6023be9b-6ea4-43d6-8d6e-6f0b1f7ed3f5_characterizeImage_855043_47.stdout parsl.dataflow.dflow INFO: Standard error for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/6023be9b-6ea4-43d6-8d6e-6f0b1f7ed3f5_characterizeImage_855043_47.stderr parsl.dataflow.dflow INFO: Standard output for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/ae16d5eb-0d3b-4e98-b48c-156ad007c5c6_isr_855043_47.stdout parsl.dataflow.dflow INFO: Standard error for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/ae16d5eb-0d3b-4e98-b48c-156ad007c5c6_isr_855043_47.stderr parsl.dataflow.dflow INFO: Task 201 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d25fb8e613b464395dcc51c808315446 with result from task 201 parsl.dataflow.dflow DEBUG: Task 201 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 202 has memoization hash 265306c6720391e7dca37cbf6dd42491 parsl.dataflow.memoization INFO: Task 202 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46817740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 202 try 0 launched on executor multi with executor id 454 parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/df792072-9c16-47c5-8de8-896e77edbed2_characterizeImage_855138_33.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/df792072-9c16-47c5-8de8-896e77edbed2_characterizeImage_855138_33.stderr parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/fd75a794-b0b4-487b-830d-2e75bcbef545_isr_855138_33.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/fd75a794-b0b4-487b-830d-2e75bcbef545_isr_855138_33.stderr parsl.dataflow.dflow INFO: Task 249 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a868213f229530899fd644027ae8d18 with result from task 249 parsl.dataflow.dflow DEBUG: Task 249 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 250 has memoization hash e997321db0d6452cae1e8f6ac6655d02 parsl.dataflow.memoization INFO: Task 250 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46836980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 250 try 0 launched on executor multi with executor id 455 parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/48a4cff3-1b16-498c-85a0-d1af0b330f73_characterizeImage_855136_36.stdout parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/48a4cff3-1b16-498c-85a0-d1af0b330f73_characterizeImage_855136_36.stderr parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/7aaa773e-1a01-4d55-9c56-be3780ab46e4_isr_855136_36.stdout parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/7aaa773e-1a01-4d55-9c56-be3780ab46e4_isr_855136_36.stderr parsl.dataflow.dflow INFO: Task 258 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57be83cb43c87bccc220cbe13a22b5bc with result from task 258 parsl.dataflow.dflow DEBUG: Task 258 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 259 has memoization hash d7910c794b8413b9641611af7e0db07b parsl.dataflow.memoization INFO: Task 259 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa468372e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 259 try 0 launched on executor multi with executor id 456 parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/27b7841f-221f-471b-8900-4c0054eb0c96_characterizeImage_855137_5.stdout parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/27b7841f-221f-471b-8900-4c0054eb0c96_characterizeImage_855137_5.stderr parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/f1c58a52-16b0-454e-8310-0f11af24cf04_isr_855137_5.stdout parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/f1c58a52-16b0-454e-8310-0f11af24cf04_isr_855137_5.stderr parsl.dataflow.dflow INFO: Task 261 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be54af2b403d98943c2d5f53a3bfb9c0 with result from task 261 parsl.dataflow.dflow DEBUG: Task 261 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 262 has memoization hash 103b49ba7bbdec178dea05e5772d6734 parsl.dataflow.memoization INFO: Task 262 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46837600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 262 try 0 launched on executor multi with executor id 457 parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/708151c8-7239-4854-bb42-5de7caa258ee_characterizeImage_855136_21.stdout parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/708151c8-7239-4854-bb42-5de7caa258ee_characterizeImage_855136_21.stderr parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/4b8613b9-dd8a-4914-af5a-a089809ab8fd_isr_855136_21.stdout parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/4b8613b9-dd8a-4914-af5a-a089809ab8fd_isr_855136_21.stderr parsl.dataflow.dflow INFO: Task 225 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 726bf038baa5a597c8cae355fac24390 with result from task 225 parsl.dataflow.dflow DEBUG: Task 225 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 226 has memoization hash d9551be45c8e136c9616da13a257d673 parsl.dataflow.memoization INFO: Task 226 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46835080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 226 try 0 launched on executor multi with executor id 458 parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/de125ba2-dcc8-47be-bc99-d9e833a1da6d_characterizeImage_855137_48.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/de125ba2-dcc8-47be-bc99-d9e833a1da6d_characterizeImage_855137_48.stderr parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/04f21561-c52e-44f8-8e04-5bdd236579d2_isr_855137_48.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/04f21561-c52e-44f8-8e04-5bdd236579d2_isr_855137_48.stderr parsl.dataflow.dflow INFO: Task 375 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a7c56066465395a26cbc1d050d3dc45 with result from task 375 parsl.dataflow.dflow DEBUG: Task 375 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 376 has memoization hash faf13cbb836b7d198a98e93b8ad37341 parsl.dataflow.memoization INFO: Task 376 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a5c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 376 try 0 launched on executor multi with executor id 459 parsl.dataflow.dflow INFO: Standard output for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/4e4de332-b56d-47b4-ab60-a131819b3bfd_characterizeImage_855042_29.stdout parsl.dataflow.dflow INFO: Standard error for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/4e4de332-b56d-47b4-ab60-a131819b3bfd_characterizeImage_855042_29.stderr parsl.dataflow.dflow INFO: Standard output for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/5f4fa116-e97e-4908-93ab-82e991f41268_isr_855042_29.stdout parsl.dataflow.dflow INFO: Standard error for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/5f4fa116-e97e-4908-93ab-82e991f41268_isr_855042_29.stderr parsl.dataflow.dflow INFO: Task 267 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc6ab4e55647b637cc37cb88bd0d8d56 with result from task 267 parsl.dataflow.dflow DEBUG: Task 267 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 268 has memoization hash c73f116277d182ffe20ddc16198ff5d3 parsl.dataflow.memoization INFO: Task 268 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46837c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 268 try 0 launched on executor multi with executor id 460 parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/fdf7df9e-8bbc-4997-8fda-09188ab365a4_characterizeImage_855136_14.stdout parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/fdf7df9e-8bbc-4997-8fda-09188ab365a4_characterizeImage_855136_14.stderr parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/9427fa18-fe7a-4764-bd02-53fff1225684_isr_855136_14.stdout parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/9427fa18-fe7a-4764-bd02-53fff1225684_isr_855136_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:01 proc_lsst.multi:146 [INFO] found job 16912284 in provider astro proc_lsst.multi INFO: found job 16912284 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 198 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a10d0f3b9d07c72750bf5a589fc16434 with result from task 198 parsl.dataflow.dflow DEBUG: Task 198 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 199 has memoization hash 1cee8ddcefc1cc1530b2729abe60d3ba parsl.dataflow.memoization INFO: Task 199 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46817420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 199 try 0 launched on executor multi with executor id 461 parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/46a95e67-16a4-41aa-9016-9e0294305b05_characterizeImage_855042_10.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/46a95e67-16a4-41aa-9016-9e0294305b05_characterizeImage_855042_10.stderr parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/71ed56e4-60aa-4d3d-9eb9-4435c1077753_isr_855042_10.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/71ed56e4-60aa-4d3d-9eb9-4435c1077753_isr_855042_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:02 proc_lsst.multi:146 [INFO] found job 16912285 in provider astro proc_lsst.multi INFO: found job 16912285 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 279 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3b71e5c17274f11355a547410444ff3 with result from task 279 parsl.dataflow.dflow DEBUG: Task 279 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 280 has memoization hash 4a117625c4e901a2c91224ee79c950d9 parsl.dataflow.memoization INFO: Task 280 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 280 try 0 launched on executor multi with executor id 462 parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/1f2d1f4d-eb13-4ac8-86c7-7ab5ed480357_characterizeImage_855044_1.stdout parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/1f2d1f4d-eb13-4ac8-86c7-7ab5ed480357_characterizeImage_855044_1.stderr parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/b354673f-214e-4c94-a1c5-9fb7ee292d0d_isr_855044_1.stdout parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/b354673f-214e-4c94-a1c5-9fb7ee292d0d_isr_855044_1.stderr parsl.dataflow.dflow INFO: Task 216 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 20ade8268ccf15a1ae8ea0f7835e0b3e with result from task 216 parsl.dataflow.dflow DEBUG: Task 216 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 217 has memoization hash 444ad249247a24837c88048c1f89fb17 parsl.dataflow.memoization INFO: Task 217 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46834720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 217 try 0 launched on executor multi with executor id 463 parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/acc6d99f-747d-4c26-9479-29b9904d49b5_characterizeImage_855043_17.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/acc6d99f-747d-4c26-9479-29b9904d49b5_characterizeImage_855043_17.stderr parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/e001ba52-f609-4c5e-877e-5e0ab69d7658_isr_855043_17.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/e001ba52-f609-4c5e-877e-5e0ab69d7658_isr_855043_17.stderr parsl.dataflow.dflow INFO: Task 213 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 85aeab72fe85f302cd73dd1eff616393 with result from task 213 parsl.dataflow.dflow DEBUG: Task 213 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 214 has memoization hash 245fd318e300db532a878985314edde1 parsl.dataflow.memoization INFO: Task 214 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46834180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 214 try 0 launched on executor multi with executor id 464 parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/e4892757-a874-46b3-ba71-56277a905abf_characterizeImage_855043_12.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/e4892757-a874-46b3-ba71-56277a905abf_characterizeImage_855043_12.stderr parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/157946d2-b278-4387-8aa5-e25c2c43dd37_isr_855043_12.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/157946d2-b278-4387-8aa5-e25c2c43dd37_isr_855043_12.stderr parsl.dataflow.dflow INFO: Task 273 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b245702c078f2b414b8f200a1f70a13 with result from task 273 parsl.dataflow.dflow DEBUG: Task 273 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 274 has memoization hash 18f851a9ceda775f158970aa00e65171 parsl.dataflow.memoization INFO: Task 274 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 274 try 0 launched on executor multi with executor id 465 parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/c85999bd-bf56-470d-9af1-a1632e95d4c2_characterizeImage_855136_26.stdout parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/c85999bd-bf56-470d-9af1-a1632e95d4c2_characterizeImage_855136_26.stderr parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/7804960a-684d-4bbf-997a-cb49efae53ba_isr_855136_26.stdout parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/7804960a-684d-4bbf-997a-cb49efae53ba_isr_855136_26.stderr parsl.dataflow.dflow INFO: Task 252 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec3caefe65632c85bfcc4198ed93234b with result from task 252 parsl.dataflow.dflow DEBUG: Task 252 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 253 has memoization hash ff1798c245d17aa52eff66ebaf62f69f parsl.dataflow.memoization INFO: Task 253 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46836ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 253 try 0 launched on executor multi with executor id 466 parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/1623e654-f989-44be-9a50-4b072d4c3ff1_characterizeImage_855043_6.stdout parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/1623e654-f989-44be-9a50-4b072d4c3ff1_characterizeImage_855043_6.stderr parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/31469ecf-567f-4283-933a-d2eb85cdc589_isr_855043_6.stdout parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/31469ecf-567f-4283-933a-d2eb85cdc589_isr_855043_6.stderr parsl.dataflow.dflow INFO: Task 195 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac2a79298b41a749d3c9922fe4687178 with result from task 195 parsl.dataflow.dflow DEBUG: Task 195 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 196 has memoization hash e8ed462420a3ef1096e500c842de4605 parsl.dataflow.memoization INFO: Task 196 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46817100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 196 try 0 launched on executor multi with executor id 467 parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/69a4ba7d-f02c-4651-b1fc-99939d0131be_characterizeImage_855136_45.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/69a4ba7d-f02c-4651-b1fc-99939d0131be_characterizeImage_855136_45.stderr parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/4e5967a7-3e73-45a6-9974-76bed61e140f_isr_855136_45.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/4e5967a7-3e73-45a6-9974-76bed61e140f_isr_855136_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:02 proc_lsst.multi:146 [INFO] found job 16912286 in provider astro proc_lsst.multi INFO: found job 16912286 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 234 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d55668be85cad1e63ac05fce0f722721 with result from task 234 parsl.dataflow.dflow DEBUG: Task 234 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 235 has memoization hash 8a6b9bae6e3268fe106dd516a45b7739 parsl.dataflow.memoization INFO: Task 235 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa468359e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 235 try 0 launched on executor multi with executor id 468 parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/4d69cde3-bc32-4836-86e1-34834fb92a43_characterizeImage_855044_16.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/4d69cde3-bc32-4836-86e1-34834fb92a43_characterizeImage_855044_16.stderr parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/68822afe-821e-4019-9439-1722d5ccb8bf_isr_855044_16.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/68822afe-821e-4019-9439-1722d5ccb8bf_isr_855044_16.stderr parsl.dataflow.dflow INFO: Task 255 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5763f630926476536e0d274d6008364e with result from task 255 parsl.dataflow.dflow DEBUG: Task 255 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 256 has memoization hash 0a8082e57c9fb477d54c293cfa3d83c6 parsl.dataflow.memoization INFO: Task 256 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46836fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 256 try 0 launched on executor multi with executor id 469 parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/33c9fa3e-f8ea-45a4-820b-42de9469e0a3_characterizeImage_855042_43.stdout parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/33c9fa3e-f8ea-45a4-820b-42de9469e0a3_characterizeImage_855042_43.stderr parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/1402e7ec-156e-46fd-8ed0-4b51f52891c3_isr_855042_43.stdout parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/1402e7ec-156e-46fd-8ed0-4b51f52891c3_isr_855042_43.stderr parsl.dataflow.dflow INFO: Task 285 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5b173e91d4d2c407e89ca7891e44766 with result from task 285 parsl.dataflow.dflow DEBUG: Task 285 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 286 has memoization hash 2bc3e5100576f3a1f83bb4d2c5925ef6 parsl.dataflow.memoization INFO: Task 286 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 286 try 0 launched on executor multi with executor id 470 parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/b8464340-e866-40dc-b19c-af5cfecf2950_characterizeImage_855137_29.stdout parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/b8464340-e866-40dc-b19c-af5cfecf2950_characterizeImage_855137_29.stderr parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/9d629c1b-93c1-47fb-a76e-d4f67543e9e4_isr_855137_29.stdout parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/9d629c1b-93c1-47fb-a76e-d4f67543e9e4_isr_855137_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:03 proc_lsst.multi:146 [INFO] found job 16912287 in provider astro proc_lsst.multi INFO: found job 16912287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 231 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46bb259300424cb595dde3417412442e with result from task 231 parsl.dataflow.dflow DEBUG: Task 231 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 232 has memoization hash a83f08ae137ede0cb48418596a95e3c6 parsl.dataflow.memoization INFO: Task 232 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa468356c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 232 try 0 launched on executor multi with executor id 471 parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/d8eec138-010f-4e97-9244-64a576585e53_characterizeImage_855044_48.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/d8eec138-010f-4e97-9244-64a576585e53_characterizeImage_855044_48.stderr parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/f8863c15-6528-4445-ad21-17fd41d9172d_isr_855044_48.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/f8863c15-6528-4445-ad21-17fd41d9172d_isr_855044_48.stderr parsl.dataflow.dflow INFO: Task 246 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77767580f0d7222b28bb1bcb6cc3bd6e with result from task 246 parsl.dataflow.dflow DEBUG: Task 246 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 247 has memoization hash eefb0091df06eb09ad1cbebbc05a0cfa parsl.dataflow.memoization INFO: Task 247 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46836660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 247 try 0 launched on executor multi with executor id 472 parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/3c763949-f455-4191-9c57-e64af0ebdb98_characterizeImage_855136_48.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/3c763949-f455-4191-9c57-e64af0ebdb98_characterizeImage_855136_48.stderr parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/79041e03-8b91-4975-a865-0fb8815f55e2_isr_855136_48.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/79041e03-8b91-4975-a865-0fb8815f55e2_isr_855136_48.stderr parsl.dataflow.dflow INFO: Task 237 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7742b34bdf789918ae6d185cd8c0a2e3 with result from task 237 parsl.dataflow.dflow DEBUG: Task 237 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 238 has memoization hash 0b720fbe683304386313306d3821ec2b parsl.dataflow.memoization INFO: Task 238 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46835d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 238 try 0 launched on executor multi with executor id 473 parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/73939ba5-429c-4b67-acb6-3db2c8aabb35_characterizeImage_855138_4.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/73939ba5-429c-4b67-acb6-3db2c8aabb35_characterizeImage_855138_4.stderr parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/dc49aeee-2306-4781-8d09-9c754c1e4564_isr_855138_4.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/dc49aeee-2306-4781-8d09-9c754c1e4564_isr_855138_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:03 proc_lsst.multi:146 [INFO] found job 16912288 in provider astro proc_lsst.multi INFO: found job 16912288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:04 proc_lsst.multi:146 [INFO] found job 16912289 in provider astro proc_lsst.multi INFO: found job 16912289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:05 proc_lsst.multi:146 [INFO] found job 16912290 in provider astro proc_lsst.multi INFO: found job 16912290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 174 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f2a468bf8c587908fea839c4bb4278cf with result from task 174 parsl.dataflow.dflow DEBUG: Task 174 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 175 has memoization hash e2227c802e71b20480100f7e665de45f parsl.dataflow.memoization INFO: Task 175 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46815b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 175 try 0 launched on executor multi with executor id 474 parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/ee502b78-df1d-4e10-9bfb-c08e2fc994d9_characterizeImage_855043_59.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/ee502b78-df1d-4e10-9bfb-c08e2fc994d9_characterizeImage_855043_59.stderr parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/90f8511f-e07a-41ac-9b26-4a1dd8a91393_isr_855043_59.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/90f8511f-e07a-41ac-9b26-4a1dd8a91393_isr_855043_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:05 proc_lsst.multi:146 [INFO] found job 16912291 in provider astro proc_lsst.multi INFO: found job 16912291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 738 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ec6f206b3f4a384b73eb6ce9d434703 with result from task 738 parsl.dataflow.dflow DEBUG: Task 738 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 739 has memoization hash a59e06baa56b12d815b4afdf1b7d2f67 parsl.dataflow.memoization INFO: Task 739 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 739 try 0 launched on executor multi with executor id 475 parsl.dataflow.dflow INFO: Standard output for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/bdc0f468-6576-4708-8241-f07f3a9c4ccf_characterizeImage_855136_50.stdout parsl.dataflow.dflow INFO: Standard error for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/bdc0f468-6576-4708-8241-f07f3a9c4ccf_characterizeImage_855136_50.stderr parsl.dataflow.dflow INFO: Standard output for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/7232fb5d-3692-4d1f-87de-14b8c4830bf4_isr_855136_50.stdout parsl.dataflow.dflow INFO: Standard error for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/7232fb5d-3692-4d1f-87de-14b8c4830bf4_isr_855136_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:06 proc_lsst.multi:146 [INFO] found job 16912292 in provider astro proc_lsst.multi INFO: found job 16912292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 696 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1ea4ada6592e3ef8d2b5c962bc5561e with result from task 696 parsl.dataflow.dflow DEBUG: Task 696 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 697 has memoization hash 9459e7f843614f2e3f577e31b619fe1c parsl.dataflow.memoization INFO: Task 697 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 697 try 0 launched on executor multi with executor id 476 parsl.dataflow.dflow INFO: Standard output for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/f40064a1-c578-4ea6-b34a-ddc66844b434_characterizeImage_855137_8.stdout parsl.dataflow.dflow INFO: Standard error for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/f40064a1-c578-4ea6-b34a-ddc66844b434_characterizeImage_855137_8.stderr parsl.dataflow.dflow INFO: Standard output for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/bc1e90fa-ba39-4716-b3df-42e4a5fdabe0_isr_855137_8.stdout parsl.dataflow.dflow INFO: Standard error for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/bc1e90fa-ba39-4716-b3df-42e4a5fdabe0_isr_855137_8.stderr parsl.dataflow.dflow INFO: Task 741 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b85cf7cfc6c7dbe6ce434153ea30d1a0 with result from task 741 parsl.dataflow.dflow DEBUG: Task 741 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 742 has memoization hash 1f33b90f3a9350a469cba1d9f9d03ffe parsl.dataflow.memoization INFO: Task 742 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 742 try 0 launched on executor multi with executor id 477 parsl.dataflow.dflow INFO: Standard output for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/fba365d7-440a-40a9-bf2f-2b1f25924869_characterizeImage_855042_50.stdout parsl.dataflow.dflow INFO: Standard error for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/fba365d7-440a-40a9-bf2f-2b1f25924869_characterizeImage_855042_50.stderr parsl.dataflow.dflow INFO: Standard output for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/42ebec5d-6b90-421d-8a33-1304b4821416_isr_855042_50.stdout parsl.dataflow.dflow INFO: Standard error for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/42ebec5d-6b90-421d-8a33-1304b4821416_isr_855042_50.stderr parsl.dataflow.dflow INFO: Task 432 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4366fcfa86fd9aa28bfd84a27f1b693 with result from task 432 parsl.dataflow.dflow DEBUG: Task 432 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 433 has memoization hash 2b86fe75cff5bd5dac6b3b6ad1729332 parsl.dataflow.memoization INFO: Task 433 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a4d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 433 try 0 launched on executor multi with executor id 478 parsl.dataflow.dflow INFO: Standard output for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/4b76bcd0-010c-47f6-ad54-c0caa89597bd_characterizeImage_855042_27.stdout parsl.dataflow.dflow INFO: Standard error for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/4b76bcd0-010c-47f6-ad54-c0caa89597bd_characterizeImage_855042_27.stderr parsl.dataflow.dflow INFO: Standard output for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/9f3da9cb-1833-4e0c-ad16-c2a20a95d29e_isr_855042_27.stdout parsl.dataflow.dflow INFO: Standard error for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/9f3da9cb-1833-4e0c-ad16-c2a20a95d29e_isr_855042_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:06 proc_lsst.multi:146 [INFO] found job 16912293 in provider astro proc_lsst.multi INFO: found job 16912293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 690 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7bcfd91e9a270afa16d4c8d07df2ff95 with result from task 690 parsl.dataflow.dflow DEBUG: Task 690 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 691 has memoization hash 03e1b7251e901b000d4e7bcd76c081f1 parsl.dataflow.memoization INFO: Task 691 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 691 try 0 launched on executor multi with executor id 479 parsl.dataflow.dflow INFO: Standard output for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/fc0420c1-dc84-47f7-bc99-cb2dca208ea2_characterizeImage_855136_20.stdout parsl.dataflow.dflow INFO: Standard error for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/fc0420c1-dc84-47f7-bc99-cb2dca208ea2_characterizeImage_855136_20.stderr parsl.dataflow.dflow INFO: Standard output for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/912b3cd1-8e42-40d7-8406-52a4d32bb5ca_isr_855136_20.stdout parsl.dataflow.dflow INFO: Standard error for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/912b3cd1-8e42-40d7-8406-52a4d32bb5ca_isr_855136_20.stderr parsl.dataflow.dflow INFO: Task 441 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69442ead886187dc030904a3b1a736df with result from task 441 parsl.dataflow.dflow DEBUG: Task 441 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 442 has memoization hash 2ab4494e1f109b7c79590b2d5bd66388 parsl.dataflow.memoization INFO: Task 442 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a6660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 442 try 0 launched on executor multi with executor id 480 parsl.dataflow.dflow INFO: Standard output for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/2721a225-c78f-4430-868d-65e1fa3907dd_characterizeImage_855138_1.stdout parsl.dataflow.dflow INFO: Standard error for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/2721a225-c78f-4430-868d-65e1fa3907dd_characterizeImage_855138_1.stderr parsl.dataflow.dflow INFO: Standard output for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/adb78243-5192-4ca2-a800-0e61041dadf5_isr_855138_1.stdout parsl.dataflow.dflow INFO: Standard error for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/adb78243-5192-4ca2-a800-0e61041dadf5_isr_855138_1.stderr parsl.dataflow.dflow INFO: Task 699 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a51831e452af65f72355761e1d43619d with result from task 699 parsl.dataflow.dflow DEBUG: Task 699 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 700 has memoization hash 6cf73c159371afcc10b888baff4fecc5 parsl.dataflow.memoization INFO: Task 700 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d0400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 700 try 0 launched on executor multi with executor id 481 parsl.dataflow.dflow INFO: Standard output for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/269e9f7c-2de1-4307-b32f-4a415026c400_characterizeImage_855043_45.stdout parsl.dataflow.dflow INFO: Standard error for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/269e9f7c-2de1-4307-b32f-4a415026c400_characterizeImage_855043_45.stderr parsl.dataflow.dflow INFO: Standard output for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/f47ebad5-a541-4173-b3dd-21e2f00260eb_isr_855043_45.stdout parsl.dataflow.dflow INFO: Standard error for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/f47ebad5-a541-4173-b3dd-21e2f00260eb_isr_855043_45.stderr parsl.dataflow.dflow INFO: Task 705 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7534e52d456b598b21a4944cfc74a5e9 with result from task 705 parsl.dataflow.dflow DEBUG: Task 705 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 706 has memoization hash e11d76bd79d978990813f5a66721df0a parsl.dataflow.memoization INFO: Task 706 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d0a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 706 try 0 launched on executor multi with executor id 482 parsl.dataflow.dflow INFO: Standard output for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/c36d0084-afcb-4ef7-a7e1-40bcbdb9d152_characterizeImage_855138_41.stdout parsl.dataflow.dflow INFO: Standard error for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/c36d0084-afcb-4ef7-a7e1-40bcbdb9d152_characterizeImage_855138_41.stderr parsl.dataflow.dflow INFO: Standard output for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/2fc4feaa-4ff2-4a62-a719-90436ef0d9b0_isr_855138_41.stdout parsl.dataflow.dflow INFO: Standard error for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/2fc4feaa-4ff2-4a62-a719-90436ef0d9b0_isr_855138_41.stderr parsl.dataflow.dflow INFO: Task 444 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d1ebb4e6c48500dec131da1bce8b113 with result from task 444 parsl.dataflow.dflow DEBUG: Task 444 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 445 has memoization hash dd6c834262be70d6be32ab34bab3681e parsl.dataflow.memoization INFO: Task 445 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a6b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 445 try 0 launched on executor multi with executor id 483 parsl.dataflow.dflow INFO: Standard output for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/8cdb98f6-cf32-47bf-905d-609b84a201a5_characterizeImage_855043_1.stdout parsl.dataflow.dflow INFO: Standard error for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/8cdb98f6-cf32-47bf-905d-609b84a201a5_characterizeImage_855043_1.stderr parsl.dataflow.dflow INFO: Standard output for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/b7e47673-7ccc-4113-8bfa-4fd66e3ab26c_isr_855043_1.stdout parsl.dataflow.dflow INFO: Standard error for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/b7e47673-7ccc-4113-8bfa-4fd66e3ab26c_isr_855043_1.stderr parsl.dataflow.dflow INFO: Task 702 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 663b05cd5835e9310d6b6472e0e9f1aa with result from task 702 parsl.dataflow.dflow DEBUG: Task 702 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 703 has memoization hash be531a8df134e094a32859e6f261d73c parsl.dataflow.memoization INFO: Task 703 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d0720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 703 try 0 launched on executor multi with executor id 484 parsl.dataflow.dflow INFO: Standard output for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/f22f06d8-3813-497a-b4c5-cd72dfa33f17_characterizeImage_855137_46.stdout parsl.dataflow.dflow INFO: Standard error for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/f22f06d8-3813-497a-b4c5-cd72dfa33f17_characterizeImage_855137_46.stderr parsl.dataflow.dflow INFO: Standard output for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/e7b06206-1ed8-4ade-9635-79b1a0527d9d_isr_855137_46.stdout parsl.dataflow.dflow INFO: Standard error for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/e7b06206-1ed8-4ade-9635-79b1a0527d9d_isr_855137_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:07 proc_lsst.multi:146 [INFO] found job 16912294 in provider astro proc_lsst.multi INFO: found job 16912294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 693 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 660ab07cf0ac316b22b01bea1311d8f0 with result from task 693 parsl.dataflow.dflow DEBUG: Task 693 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 694 has memoization hash 14bb3193fc8d0388d13bef9c5208908e parsl.dataflow.memoization INFO: Task 694 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 694 try 0 launched on executor multi with executor id 485 parsl.dataflow.dflow INFO: Standard output for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/a29a532a-91c5-4329-b4b6-0f860f45ef50_characterizeImage_855138_29.stdout parsl.dataflow.dflow INFO: Standard error for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/a29a532a-91c5-4329-b4b6-0f860f45ef50_characterizeImage_855138_29.stderr parsl.dataflow.dflow INFO: Standard output for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/21d1c744-f237-40b4-8f3f-6af59d76f710_isr_855138_29.stdout parsl.dataflow.dflow INFO: Standard error for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/21d1c744-f237-40b4-8f3f-6af59d76f710_isr_855138_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:08 proc_lsst.multi:146 [INFO] found job 16912295 in provider astro proc_lsst.multi INFO: found job 16912295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 420 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d960c5fa05d444e71aeacc2848f0b272 with result from task 420 parsl.dataflow.dflow DEBUG: Task 420 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 421 has memoization hash dfdaed7036a7245c025a626ae79b9dbe parsl.dataflow.memoization INFO: Task 421 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a4a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 421 try 0 launched on executor multi with executor id 486 parsl.dataflow.dflow INFO: Standard output for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/ca326c19-8625-4753-8a7c-5e60e09448fb_characterizeImage_855136_28.stdout parsl.dataflow.dflow INFO: Standard error for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/ca326c19-8625-4753-8a7c-5e60e09448fb_characterizeImage_855136_28.stderr parsl.dataflow.dflow INFO: Standard output for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/7bbeab17-91cd-4a10-a2e9-0a88c4fb4a71_isr_855136_28.stdout parsl.dataflow.dflow INFO: Standard error for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/7bbeab17-91cd-4a10-a2e9-0a88c4fb4a71_isr_855136_28.stderr parsl.dataflow.dflow INFO: Task 747 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86d7074255b197fe6e78d9ca393fb10b with result from task 747 parsl.dataflow.dflow DEBUG: Task 747 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 748 has memoization hash 0076747ec9709b221100d2c431ceba3a parsl.dataflow.memoization INFO: Task 748 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 748 try 0 launched on executor multi with executor id 487 parsl.dataflow.dflow INFO: Standard output for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/29a47264-15d8-4b28-a875-b43a0635e26f_characterizeImage_855136_31.stdout parsl.dataflow.dflow INFO: Standard error for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/29a47264-15d8-4b28-a875-b43a0635e26f_characterizeImage_855136_31.stderr parsl.dataflow.dflow INFO: Standard output for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/cdedcd16-ab0e-4f6d-a5c8-13cc762eb183_isr_855136_31.stdout parsl.dataflow.dflow INFO: Standard error for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/cdedcd16-ab0e-4f6d-a5c8-13cc762eb183_isr_855136_31.stderr parsl.dataflow.dflow INFO: Task 405 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bcaa42173e72e319b330baa3cfb8fb9f with result from task 405 parsl.dataflow.dflow DEBUG: Task 405 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 406 has memoization hash aab2e39a471790184c4908d49de76a37 parsl.dataflow.memoization INFO: Task 406 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a74c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 406 try 0 launched on executor multi with executor id 488 parsl.dataflow.dflow INFO: Standard output for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/f93ccfa3-d707-4b23-ab51-6ccbad917d52_characterizeImage_855137_45.stdout parsl.dataflow.dflow INFO: Standard error for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/f93ccfa3-d707-4b23-ab51-6ccbad917d52_characterizeImage_855137_45.stderr parsl.dataflow.dflow INFO: Standard output for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/2a0f9084-0715-4aba-8314-243abbb1f521_isr_855137_45.stdout parsl.dataflow.dflow INFO: Standard error for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/2a0f9084-0715-4aba-8314-243abbb1f521_isr_855137_45.stderr parsl.dataflow.dflow INFO: Task 435 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c58a30940c8723320836483b892c64c4 with result from task 435 parsl.dataflow.dflow DEBUG: Task 435 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 436 has memoization hash d655dc835f9a48e141c3d929ad27c0c1 parsl.dataflow.memoization INFO: Task 436 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a60c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 436 try 0 launched on executor multi with executor id 489 parsl.dataflow.dflow INFO: Standard output for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/639f6a4e-87dc-4d83-93b8-dea0bafbadc6_characterizeImage_855137_14.stdout parsl.dataflow.dflow INFO: Standard error for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/639f6a4e-87dc-4d83-93b8-dea0bafbadc6_characterizeImage_855137_14.stderr parsl.dataflow.dflow INFO: Standard output for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/55a1bd20-2fe8-4b30-ac1c-ceb79006b994_isr_855137_14.stdout parsl.dataflow.dflow INFO: Standard error for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/55a1bd20-2fe8-4b30-ac1c-ceb79006b994_isr_855137_14.stderr parsl.dataflow.dflow INFO: Task 708 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 262e9747133968a1939157ca46e849c5 with result from task 708 parsl.dataflow.dflow DEBUG: Task 708 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 709 has memoization hash 3516822fe2d377a91847afb9f42b05ee parsl.dataflow.memoization INFO: Task 709 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d0d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 709 try 0 launched on executor multi with executor id 490 parsl.dataflow.dflow INFO: Standard output for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/8ca427e1-013e-47bd-907f-6b8663f7cf67_characterizeImage_855042_5.stdout parsl.dataflow.dflow INFO: Standard error for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/8ca427e1-013e-47bd-907f-6b8663f7cf67_characterizeImage_855042_5.stderr parsl.dataflow.dflow INFO: Standard output for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/148889c1-b5c5-4710-8aad-58e58c04b460_isr_855042_5.stdout parsl.dataflow.dflow INFO: Standard error for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/148889c1-b5c5-4710-8aad-58e58c04b460_isr_855042_5.stderr parsl.dataflow.dflow INFO: Task 426 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e206d2b3846cb3cb5439739613676aa8 with result from task 426 parsl.dataflow.dflow DEBUG: Task 426 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 427 has memoization hash 4332d370b193ab64500c63fa1a356635 parsl.dataflow.memoization INFO: Task 427 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a53a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 427 try 0 launched on executor multi with executor id 491 parsl.dataflow.dflow INFO: Standard output for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/2465daeb-2c94-470c-af3c-eaed21ce6fa4_characterizeImage_855136_25.stdout parsl.dataflow.dflow INFO: Standard error for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/2465daeb-2c94-470c-af3c-eaed21ce6fa4_characterizeImage_855136_25.stderr parsl.dataflow.dflow INFO: Standard output for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/0127fa18-ffa9-423a-96ed-d782e7ec314f_isr_855136_25.stdout parsl.dataflow.dflow INFO: Standard error for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/0127fa18-ffa9-423a-96ed-d782e7ec314f_isr_855136_25.stderr parsl.dataflow.dflow INFO: Task 447 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 753a02baa1f6067cad4894dedf6495d7 with result from task 447 parsl.dataflow.dflow DEBUG: Task 447 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 448 has memoization hash 73ca16f325173dc87d5774815473ceb9 parsl.dataflow.memoization INFO: Task 448 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a67a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 448 try 0 launched on executor multi with executor id 492 parsl.dataflow.dflow INFO: Standard output for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/3dc52591-ef88-4e62-89d4-0bd89989e2fa_characterizeImage_855137_49.stdout parsl.dataflow.dflow INFO: Standard error for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/3dc52591-ef88-4e62-89d4-0bd89989e2fa_characterizeImage_855137_49.stderr parsl.dataflow.dflow INFO: Standard output for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/47867cb3-4d13-43de-b532-7cf719e45c2f_isr_855137_49.stdout parsl.dataflow.dflow INFO: Standard error for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/47867cb3-4d13-43de-b532-7cf719e45c2f_isr_855137_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:09 proc_lsst.multi:146 [INFO] found job 16912296 in provider astro proc_lsst.multi INFO: found job 16912296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 402 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 04ea167d93e64eceafa0ffea65cbcc5c with result from task 402 parsl.dataflow.dflow DEBUG: Task 402 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 403 has memoization hash f8a8b77e4699db74c29fbf3108fd5c80 parsl.dataflow.memoization INFO: Task 403 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a77e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 403 try 0 launched on executor multi with executor id 493 parsl.dataflow.dflow INFO: Standard output for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/e91e187b-a720-4615-863e-9439f8e3cba9_characterizeImage_855136_43.stdout parsl.dataflow.dflow INFO: Standard error for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/e91e187b-a720-4615-863e-9439f8e3cba9_characterizeImage_855136_43.stderr parsl.dataflow.dflow INFO: Standard output for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/c6897742-65df-49f4-b23a-ea76e7ec1327_isr_855136_43.stdout parsl.dataflow.dflow INFO: Standard error for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/c6897742-65df-49f4-b23a-ea76e7ec1327_isr_855136_43.stderr parsl.dataflow.dflow INFO: Task 429 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6574ea7554dccaa14de3ab8775118c2a with result from task 429 parsl.dataflow.dflow DEBUG: Task 429 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 430 has memoization hash 1be0d67a211de1f24d4a8973f357eaab parsl.dataflow.memoization INFO: Task 430 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a5080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 430 try 0 launched on executor multi with executor id 494 parsl.dataflow.dflow INFO: Standard output for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/b3cec1e7-ef40-4f00-b70b-125a3614f4c8_characterizeImage_855043_4.stdout parsl.dataflow.dflow INFO: Standard error for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/b3cec1e7-ef40-4f00-b70b-125a3614f4c8_characterizeImage_855043_4.stderr parsl.dataflow.dflow INFO: Standard output for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/70d759b5-0af7-441a-9f2b-dbc045f69a39_isr_855043_4.stdout parsl.dataflow.dflow INFO: Standard error for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/70d759b5-0af7-441a-9f2b-dbc045f69a39_isr_855043_4.stderr parsl.dataflow.dflow INFO: Task 735 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b3154c70555747d2e9ca482ff7fa8f2 with result from task 735 parsl.dataflow.dflow DEBUG: Task 735 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 736 has memoization hash 29faed971da8e617fc962788950e23d5 parsl.dataflow.memoization INFO: Task 736 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 736 try 0 launched on executor multi with executor id 495 parsl.dataflow.dflow INFO: Standard output for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/360351ff-fc5d-4d3b-85f7-440d48d8eafd_characterizeImage_855044_6.stdout parsl.dataflow.dflow INFO: Standard error for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/360351ff-fc5d-4d3b-85f7-440d48d8eafd_characterizeImage_855044_6.stderr parsl.dataflow.dflow INFO: Standard output for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/aa33b260-f642-4272-8ad4-9a20ca12f149_isr_855044_6.stdout parsl.dataflow.dflow INFO: Standard error for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/aa33b260-f642-4272-8ad4-9a20ca12f149_isr_855044_6.stderr parsl.dataflow.dflow INFO: Task 438 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa5076301b92ac5f87297b713edffc8f with result from task 438 parsl.dataflow.dflow DEBUG: Task 438 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 439 has memoization hash 9f98d9e78d3498b9222f410447cb2f0d parsl.dataflow.memoization INFO: Task 439 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a6160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 439 try 0 launched on executor multi with executor id 496 parsl.dataflow.dflow INFO: Standard output for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/2edc0738-eb77-40ff-a934-6a8b6bf2b91d_characterizeImage_855044_23.stdout parsl.dataflow.dflow INFO: Standard error for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/2edc0738-eb77-40ff-a934-6a8b6bf2b91d_characterizeImage_855044_23.stderr parsl.dataflow.dflow INFO: Standard output for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/ae8ce82a-fa7c-41ed-ab45-fb37599feb99_isr_855044_23.stdout parsl.dataflow.dflow INFO: Standard error for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/ae8ce82a-fa7c-41ed-ab45-fb37599feb99_isr_855044_23.stderr parsl.dataflow.dflow INFO: Task 411 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a2eb5e56d261937a166ccf22f341397 with result from task 411 parsl.dataflow.dflow DEBUG: Task 411 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 412 has memoization hash 79230e1a5ec12d5c630eefa3c05daab9 parsl.dataflow.memoization INFO: Task 412 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a6e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 412 try 0 launched on executor multi with executor id 497 parsl.dataflow.dflow INFO: Standard output for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/693466bf-af0d-4d00-904d-d5d302fc91e1_characterizeImage_855136_38.stdout parsl.dataflow.dflow INFO: Standard error for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/693466bf-af0d-4d00-904d-d5d302fc91e1_characterizeImage_855136_38.stderr parsl.dataflow.dflow INFO: Standard output for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/f164302b-5e1b-4175-97fd-b1e19adf0f1b_isr_855136_38.stdout parsl.dataflow.dflow INFO: Standard error for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/f164302b-5e1b-4175-97fd-b1e19adf0f1b_isr_855136_38.stderr parsl.dataflow.dflow INFO: Task 750 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc6b34532dfbe0384e30df5aaad7d326 with result from task 750 parsl.dataflow.dflow DEBUG: Task 750 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 751 has memoization hash a3d73041bd71b12124a9e9355188758c parsl.dataflow.memoization INFO: Task 751 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 751 try 0 launched on executor multi with executor id 498 parsl.dataflow.dflow INFO: Standard output for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/9702e711-d34d-4709-ad99-42b9d440be0b_characterizeImage_855138_44.stdout parsl.dataflow.dflow INFO: Standard error for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/9702e711-d34d-4709-ad99-42b9d440be0b_characterizeImage_855138_44.stderr parsl.dataflow.dflow INFO: Standard output for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/63ce4282-0e9c-4f93-a841-61ff9befa86f_isr_855138_44.stdout parsl.dataflow.dflow INFO: Standard error for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/63ce4282-0e9c-4f93-a841-61ff9befa86f_isr_855138_44.stderr parsl.dataflow.dflow INFO: Task 759 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f68f7fd65b1180ee9d5850e584fcbf5 with result from task 759 parsl.dataflow.dflow DEBUG: Task 759 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 760 has memoization hash 7e7743679a34288a3d50fe35305d3b76 parsl.dataflow.memoization INFO: Task 760 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d3240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 760 try 0 launched on executor multi with executor id 499 parsl.dataflow.dflow INFO: Standard output for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/6a8ffdf0-a4fc-44dd-8e7b-1cf29dadc7fe_characterizeImage_855043_21.stdout parsl.dataflow.dflow INFO: Standard error for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/6a8ffdf0-a4fc-44dd-8e7b-1cf29dadc7fe_characterizeImage_855043_21.stderr parsl.dataflow.dflow INFO: Standard output for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/120b868d-ca98-4700-8835-578eeb788d39_isr_855043_21.stdout parsl.dataflow.dflow INFO: Standard error for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/120b868d-ca98-4700-8835-578eeb788d39_isr_855043_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:09 proc_lsst.multi:146 [INFO] found job 16912297 in provider astro proc_lsst.multi INFO: found job 16912297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 408 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a15e8b5673c82965c2899df2b2d84126 with result from task 408 parsl.dataflow.dflow DEBUG: Task 408 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 409 has memoization hash b20a6050b61074153fc65514209be77d parsl.dataflow.memoization INFO: Task 409 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a71a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 409 try 0 launched on executor multi with executor id 500 parsl.dataflow.dflow INFO: Standard output for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/b4179868-7eca-4d3f-a9cf-cb6b3cd82fdf_characterizeImage_855043_48.stdout parsl.dataflow.dflow INFO: Standard error for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/b4179868-7eca-4d3f-a9cf-cb6b3cd82fdf_characterizeImage_855043_48.stderr parsl.dataflow.dflow INFO: Standard output for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/ce4cd375-67f5-4bfd-ab0f-c9c57797dae2_isr_855043_48.stdout parsl.dataflow.dflow INFO: Standard error for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/ce4cd375-67f5-4bfd-ab0f-c9c57797dae2_isr_855043_48.stderr parsl.dataflow.dflow INFO: Task 414 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 99bf0eccd0b9896a5e192ffd6463babb with result from task 414 parsl.dataflow.dflow DEBUG: Task 414 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 415 has memoization hash 705f2243b677898262b839c1dbdeb047 parsl.dataflow.memoization INFO: Task 415 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a4540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 415 try 0 launched on executor multi with executor id 501 parsl.dataflow.dflow INFO: Standard output for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/b2bd430f-a50f-44fc-a842-856b57745a76_characterizeImage_855042_12.stdout parsl.dataflow.dflow INFO: Standard error for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/b2bd430f-a50f-44fc-a842-856b57745a76_characterizeImage_855042_12.stderr parsl.dataflow.dflow INFO: Standard output for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/2c565da0-50df-4160-b487-d12b1ed06ff4_isr_855042_12.stdout parsl.dataflow.dflow INFO: Standard error for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/2c565da0-50df-4160-b487-d12b1ed06ff4_isr_855042_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:10 proc_lsst.multi:146 [INFO] found job 16912298 in provider astro proc_lsst.multi INFO: found job 16912298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:11 proc_lsst.multi:146 [INFO] found job 16912299 in provider astro proc_lsst.multi INFO: found job 16912299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 756 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8084bc3f8f11c9a85b913ab5e7c6771c with result from task 756 parsl.dataflow.dflow DEBUG: Task 756 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 757 has memoization hash d02daa61d8df9a1c47f43c766a1b1941 parsl.dataflow.memoization INFO: Task 757 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d2f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 757 try 0 launched on executor multi with executor id 502 parsl.dataflow.dflow INFO: Standard output for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/c216f620-b82e-4520-ade0-c0f03b24f500_characterizeImage_855044_10.stdout parsl.dataflow.dflow INFO: Standard error for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/c216f620-b82e-4520-ade0-c0f03b24f500_characterizeImage_855044_10.stderr parsl.dataflow.dflow INFO: Standard output for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/b51a2d01-4820-46b1-9415-c0e7abde6f81_isr_855044_10.stdout parsl.dataflow.dflow INFO: Standard error for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/b51a2d01-4820-46b1-9415-c0e7abde6f81_isr_855044_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:12 proc_lsst.multi:146 [INFO] found job 16912300 in provider astro proc_lsst.multi INFO: found job 16912300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 768 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6eacaf42c3904455cc5bf643ba96f4f3 with result from task 768 parsl.dataflow.dflow DEBUG: Task 768 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 769 has memoization hash 9425327743364f7cdf475b1a2d18afb1 parsl.dataflow.memoization INFO: Task 769 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d3c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 769 try 0 launched on executor multi with executor id 503 parsl.dataflow.dflow INFO: Standard output for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/1a3e5edb-849f-4780-9395-b6795a89e8f1_characterizeImage_855137_1.stdout parsl.dataflow.dflow INFO: Standard error for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/1a3e5edb-849f-4780-9395-b6795a89e8f1_characterizeImage_855137_1.stderr parsl.dataflow.dflow INFO: Standard output for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/41fc44e2-dbaa-482c-aea5-68f5974f63ea_isr_855137_1.stdout parsl.dataflow.dflow INFO: Standard error for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/41fc44e2-dbaa-482c-aea5-68f5974f63ea_isr_855137_1.stderr parsl.dataflow.dflow INFO: Task 765 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 317d47b21d3d1eb013e7d530872078f4 with result from task 765 parsl.dataflow.dflow DEBUG: Task 765 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 766 has memoization hash 3f285534a51b38b20083bc1c8b1b12fe parsl.dataflow.memoization INFO: Task 766 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d3920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 766 try 0 launched on executor multi with executor id 504 parsl.dataflow.dflow INFO: Standard output for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/d4e88c59-8f47-4eae-9c37-a3daa4745c40_characterizeImage_855137_44.stdout parsl.dataflow.dflow INFO: Standard error for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/d4e88c59-8f47-4eae-9c37-a3daa4745c40_characterizeImage_855137_44.stderr parsl.dataflow.dflow INFO: Standard output for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/a8dd069a-1d3d-408b-a849-493c6e4560c8_isr_855137_44.stdout parsl.dataflow.dflow INFO: Standard error for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/a8dd069a-1d3d-408b-a849-493c6e4560c8_isr_855137_44.stderr parsl.dataflow.dflow INFO: Task 744 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57651aa18b741db85f7af3509bf5594f with result from task 744 parsl.dataflow.dflow DEBUG: Task 744 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 745 has memoization hash 051af44c100be5b28bd9f9b9f6a29eca parsl.dataflow.memoization INFO: Task 745 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 745 try 0 launched on executor multi with executor id 505 parsl.dataflow.dflow INFO: Standard output for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/8b3a159d-41d3-4144-94d4-216a47942b23_characterizeImage_855043_15.stdout parsl.dataflow.dflow INFO: Standard error for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/8b3a159d-41d3-4144-94d4-216a47942b23_characterizeImage_855043_15.stderr parsl.dataflow.dflow INFO: Standard output for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/38be518b-ab5f-41ca-ab76-5778085f0ebe_isr_855043_15.stdout parsl.dataflow.dflow INFO: Standard error for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/38be518b-ab5f-41ca-ab76-5778085f0ebe_isr_855043_15.stderr parsl.dataflow.dflow INFO: Task 753 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d4de956d9df5cc3781e7b459ff7d5d05 with result from task 753 parsl.dataflow.dflow DEBUG: Task 753 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 754 has memoization hash 1e5c7152c8795042dcb221a1c8f08cc9 parsl.dataflow.memoization INFO: Task 754 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d2b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 754 try 0 launched on executor multi with executor id 506 parsl.dataflow.dflow INFO: Standard output for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/aa03a199-289f-476e-900a-c9824b086ad5_characterizeImage_855042_47.stdout parsl.dataflow.dflow INFO: Standard error for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/aa03a199-289f-476e-900a-c9824b086ad5_characterizeImage_855042_47.stderr parsl.dataflow.dflow INFO: Standard output for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/7a29285a-9903-4c78-a08b-ed50e8d9a14d_isr_855042_47.stdout parsl.dataflow.dflow INFO: Standard error for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/7a29285a-9903-4c78-a08b-ed50e8d9a14d_isr_855042_47.stderr parsl.dataflow.dflow INFO: Task 582 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6301c908fb54ecc31ffb69e8d48ab514 with result from task 582 parsl.dataflow.dflow DEBUG: Task 582 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 583 has memoization hash a329a61a85043143edb37ebe69966713 parsl.dataflow.memoization INFO: Task 583 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa457431a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 583 try 0 launched on executor multi with executor id 507 parsl.dataflow.dflow INFO: Standard output for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/7900b214-4a80-4deb-9c2d-7b0fa7cff3fb_characterizeImage_855042_57.stdout parsl.dataflow.dflow INFO: Standard error for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/7900b214-4a80-4deb-9c2d-7b0fa7cff3fb_characterizeImage_855042_57.stderr parsl.dataflow.dflow INFO: Standard output for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/cb3598df-94bf-4dc0-b56f-1a2bb41f6df0_isr_855042_57.stdout parsl.dataflow.dflow INFO: Standard error for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/cb3598df-94bf-4dc0-b56f-1a2bb41f6df0_isr_855042_57.stderr parsl.dataflow.dflow INFO: Task 801 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7716a9be9b39b8ebfbe02013f15514b3 with result from task 801 parsl.dataflow.dflow DEBUG: Task 801 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 802 has memoization hash f194a5b3aed37b05e780c4da5d373ee1 parsl.dataflow.memoization INFO: Task 802 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 802 try 0 launched on executor multi with executor id 508 parsl.dataflow.dflow INFO: Standard output for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/476a5fdb-6792-4fc0-8b21-94ef66d94625_characterizeImage_855138_2.stdout parsl.dataflow.dflow INFO: Standard error for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/476a5fdb-6792-4fc0-8b21-94ef66d94625_characterizeImage_855138_2.stderr parsl.dataflow.dflow INFO: Standard output for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/0e0d23c0-3d5f-4c6e-920d-9ddd40b6ba3a_isr_855138_2.stdout parsl.dataflow.dflow INFO: Standard error for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/0e0d23c0-3d5f-4c6e-920d-9ddd40b6ba3a_isr_855138_2.stderr parsl.dataflow.dflow INFO: Task 585 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a4dba9c467beb17fb486084c66609e6 with result from task 585 parsl.dataflow.dflow DEBUG: Task 585 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 586 has memoization hash 918255a4edcba52134f51317bf1e21ee parsl.dataflow.memoization INFO: Task 586 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45742e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 586 try 0 launched on executor multi with executor id 509 parsl.dataflow.dflow INFO: Standard output for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/dc375654-3833-4292-bcce-6af3cad4d0eb_characterizeImage_855042_21.stdout parsl.dataflow.dflow INFO: Standard error for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/dc375654-3833-4292-bcce-6af3cad4d0eb_characterizeImage_855042_21.stderr parsl.dataflow.dflow INFO: Standard output for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/ea0bd3f8-9f3f-48df-93aa-87731f39acf1_isr_855042_21.stdout parsl.dataflow.dflow INFO: Standard error for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/ea0bd3f8-9f3f-48df-93aa-87731f39acf1_isr_855042_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:15 proc_lsst.multi:146 [INFO] found job 16912301 in provider astro proc_lsst.multi INFO: found job 16912301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 591 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 243bc388a8aa6faff66b92ac85a7ea01 with result from task 591 parsl.dataflow.dflow DEBUG: Task 591 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 592 has memoization hash 9cb5010a0bec05b8824b186945ab83b7 parsl.dataflow.memoization INFO: Task 592 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa457404a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 592 try 0 launched on executor multi with executor id 510 parsl.dataflow.dflow INFO: Standard output for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/6dc1d204-da4f-4d16-9ae3-f8c7f465049a_characterizeImage_855138_43.stdout parsl.dataflow.dflow INFO: Standard error for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/6dc1d204-da4f-4d16-9ae3-f8c7f465049a_characterizeImage_855138_43.stderr parsl.dataflow.dflow INFO: Standard output for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/e801a22d-2413-4ad6-b34d-d75f67ee8544_isr_855138_43.stdout parsl.dataflow.dflow INFO: Standard error for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/e801a22d-2413-4ad6-b34d-d75f67ee8544_isr_855138_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:15 proc_lsst.multi:146 [INFO] found job 16912302 in provider astro proc_lsst.multi INFO: found job 16912302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 573 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4029a0046027ad0652898adedc140952 with result from task 573 parsl.dataflow.dflow DEBUG: Task 573 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 574 has memoization hash c7b008856ddd413dbf03ad72b12fffe4 parsl.dataflow.memoization INFO: Task 574 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45702de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 574 try 0 launched on executor multi with executor id 511 parsl.dataflow.dflow INFO: Standard output for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/d7dac109-b10f-437b-a91d-3cc7ded677f2_characterizeImage_855138_6.stdout parsl.dataflow.dflow INFO: Standard error for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/d7dac109-b10f-437b-a91d-3cc7ded677f2_characterizeImage_855138_6.stderr parsl.dataflow.dflow INFO: Standard output for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/6ba834bb-b634-4c79-a5fa-31efd7af0028_isr_855138_6.stdout parsl.dataflow.dflow INFO: Standard error for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/6ba834bb-b634-4c79-a5fa-31efd7af0028_isr_855138_6.stderr parsl.dataflow.dflow INFO: Task 570 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f0ae18f4d80f49cdffe1acb27bd7c52e with result from task 570 parsl.dataflow.dflow DEBUG: Task 570 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 571 has memoization hash d51d33fab9285dd100640c93e94f2e0d parsl.dataflow.memoization INFO: Task 571 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45702fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 571 try 0 launched on executor multi with executor id 512 parsl.dataflow.dflow INFO: Standard output for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/6662812e-ff40-4ff7-9759-8f7e94d0c502_characterizeImage_855043_53.stdout parsl.dataflow.dflow INFO: Standard error for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/6662812e-ff40-4ff7-9759-8f7e94d0c502_characterizeImage_855043_53.stderr parsl.dataflow.dflow INFO: Standard output for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/16673fa5-4f22-4d84-82de-aea6d550cfe5_isr_855043_53.stdout parsl.dataflow.dflow INFO: Standard error for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/16673fa5-4f22-4d84-82de-aea6d550cfe5_isr_855043_53.stderr parsl.dataflow.dflow INFO: Task 588 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d550ea72389fd5af3bc66e24059db2ad with result from task 588 parsl.dataflow.dflow DEBUG: Task 588 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 589 has memoization hash 161fb25842ac90a8260b2ebb90b4d95e parsl.dataflow.memoization INFO: Task 589 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45742b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 589 try 0 launched on executor multi with executor id 513 parsl.dataflow.dflow INFO: Standard output for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/52372e7a-1d04-440d-b649-95dac62cc2a7_characterizeImage_855042_48.stdout parsl.dataflow.dflow INFO: Standard error for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/52372e7a-1d04-440d-b649-95dac62cc2a7_characterizeImage_855042_48.stderr parsl.dataflow.dflow INFO: Standard output for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/72bf7e26-d949-4292-a8a3-fc02c148777c_isr_855042_48.stdout parsl.dataflow.dflow INFO: Standard error for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/72bf7e26-d949-4292-a8a3-fc02c148777c_isr_855042_48.stderr parsl.dataflow.dflow INFO: Task 576 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f0ea5cd7576695fcc736844aed65e1c3 with result from task 576 parsl.dataflow.dflow DEBUG: Task 576 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 577 has memoization hash f91764a5e09d442e62b2f2f9cdd420b1 parsl.dataflow.memoization INFO: Task 577 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45700ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 577 try 0 launched on executor multi with executor id 514 parsl.dataflow.dflow INFO: Standard output for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/e8e15fd7-29b9-4789-ae3d-21b93a64ec0c_characterizeImage_855044_18.stdout parsl.dataflow.dflow INFO: Standard error for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/e8e15fd7-29b9-4789-ae3d-21b93a64ec0c_characterizeImage_855044_18.stderr parsl.dataflow.dflow INFO: Standard output for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/3955e772-5af4-4770-b169-e3203ee81886_isr_855044_18.stdout parsl.dataflow.dflow INFO: Standard error for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/3955e772-5af4-4770-b169-e3203ee81886_isr_855044_18.stderr parsl.dataflow.dflow INFO: Task 642 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b178a77d1c05747f015df355457507e2 with result from task 642 parsl.dataflow.dflow DEBUG: Task 642 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 643 has memoization hash ffb841b5f269d82827892b35c99b7b6c parsl.dataflow.memoization INFO: Task 643 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 643 try 0 launched on executor multi with executor id 515 parsl.dataflow.dflow INFO: Standard output for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/05450563-8e9d-4013-8aa9-dd2e2e248f33_characterizeImage_855043_33.stdout parsl.dataflow.dflow INFO: Standard error for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/05450563-8e9d-4013-8aa9-dd2e2e248f33_characterizeImage_855043_33.stderr parsl.dataflow.dflow INFO: Standard output for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/69674bdc-bf39-4848-8f3e-c14dd097ba05_isr_855043_33.stdout parsl.dataflow.dflow INFO: Standard error for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/69674bdc-bf39-4848-8f3e-c14dd097ba05_isr_855043_33.stderr parsl.dataflow.dflow INFO: Task 651 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc4fb17ec2240c520a693d572fe013c9 with result from task 651 parsl.dataflow.dflow DEBUG: Task 651 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 652 has memoization hash 7763ec8ffa4d72f3f38b17b5a07a4c0b parsl.dataflow.memoization INFO: Task 652 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 652 try 0 launched on executor multi with executor id 516 parsl.dataflow.dflow INFO: Standard output for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/89538e24-b316-4485-94df-162c768c72f1_characterizeImage_855042_9.stdout parsl.dataflow.dflow INFO: Standard error for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/89538e24-b316-4485-94df-162c768c72f1_characterizeImage_855042_9.stderr parsl.dataflow.dflow INFO: Standard output for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/54fe1f45-3afe-4585-8d7a-1c4cb6d6f838_isr_855042_9.stdout parsl.dataflow.dflow INFO: Standard error for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/54fe1f45-3afe-4585-8d7a-1c4cb6d6f838_isr_855042_9.stderr parsl.dataflow.dflow INFO: Task 654 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38a1278a0a896541bcbc2f8d63d75f4c with result from task 654 parsl.dataflow.dflow DEBUG: Task 654 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 655 has memoization hash 43a39929fec5d01bd6422e4bb576bfd6 parsl.dataflow.memoization INFO: Task 655 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 655 try 0 launched on executor multi with executor id 517 parsl.dataflow.dflow INFO: Standard output for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/5b3e887d-5921-44e4-802b-329643e5dc09_characterizeImage_855043_39.stdout parsl.dataflow.dflow INFO: Standard error for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/5b3e887d-5921-44e4-802b-329643e5dc09_characterizeImage_855043_39.stderr parsl.dataflow.dflow INFO: Standard output for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/dddc8689-aa67-48f7-8162-5be429c46489_isr_855043_39.stdout parsl.dataflow.dflow INFO: Standard error for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/dddc8689-aa67-48f7-8162-5be429c46489_isr_855043_39.stderr parsl.dataflow.dflow INFO: Task 663 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e9e3fab5891d2a71b98c09a7aebec164 with result from task 663 parsl.dataflow.dflow DEBUG: Task 663 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 664 has memoization hash a27fb6b673dde82ec26048609a1892eb parsl.dataflow.memoization INFO: Task 664 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 664 try 0 launched on executor multi with executor id 518 parsl.dataflow.dflow INFO: Standard output for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/59653cd9-1bdd-412d-a9b9-422a8195672a_characterizeImage_855042_18.stdout parsl.dataflow.dflow INFO: Standard error for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/59653cd9-1bdd-412d-a9b9-422a8195672a_characterizeImage_855042_18.stderr parsl.dataflow.dflow INFO: Standard output for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/d73c9bea-e591-48de-8e35-483cae672d65_isr_855042_18.stdout parsl.dataflow.dflow INFO: Standard error for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/d73c9bea-e591-48de-8e35-483cae672d65_isr_855042_18.stderr parsl.dataflow.dflow INFO: Task 645 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 176f8e6b8540acb364a73f89de1d2e95 with result from task 645 parsl.dataflow.dflow DEBUG: Task 645 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 646 has memoization hash bfe2266b5d9ce3fc4f0d27379d2d8574 parsl.dataflow.memoization INFO: Task 646 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 646 try 0 launched on executor multi with executor id 519 parsl.dataflow.dflow INFO: Standard output for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/dd732501-883e-4c66-952e-f5ffa4d0ee73_characterizeImage_855043_44.stdout parsl.dataflow.dflow INFO: Standard error for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/dd732501-883e-4c66-952e-f5ffa4d0ee73_characterizeImage_855043_44.stderr parsl.dataflow.dflow INFO: Standard output for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/3f6102e3-4418-4a2d-9fd9-1be09135cae9_isr_855043_44.stdout parsl.dataflow.dflow INFO: Standard error for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/3f6102e3-4418-4a2d-9fd9-1be09135cae9_isr_855043_44.stderr parsl.dataflow.dflow INFO: Task 648 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33a4d45a6d0ce91cb890a6f40445753c with result from task 648 parsl.dataflow.dflow DEBUG: Task 648 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 649 has memoization hash a6fbf30db7d8d084ed87f82579f43b0f parsl.dataflow.memoization INFO: Task 649 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 649 try 0 launched on executor multi with executor id 520 parsl.dataflow.dflow INFO: Standard output for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/670e7359-a626-40a5-b51f-69d14a21fb27_characterizeImage_855137_26.stdout parsl.dataflow.dflow INFO: Standard error for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/670e7359-a626-40a5-b51f-69d14a21fb27_characterizeImage_855137_26.stderr parsl.dataflow.dflow INFO: Standard output for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/b85d4624-a0b6-4f9b-8cbb-a4b5f1b2375c_isr_855137_26.stdout parsl.dataflow.dflow INFO: Standard error for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/b85d4624-a0b6-4f9b-8cbb-a4b5f1b2375c_isr_855137_26.stderr parsl.dataflow.dflow INFO: Task 555 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 15e86939a75f5a96588e46a78b9026d5 with result from task 555 parsl.dataflow.dflow DEBUG: Task 555 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 556 has memoization hash 58790f9b093f0b26747f8b2591980834 parsl.dataflow.memoization INFO: Task 556 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45702340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 556 try 0 launched on executor multi with executor id 521 parsl.dataflow.dflow INFO: Standard output for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/81ecc660-121d-4c7d-bbbb-ff626d43f903_characterizeImage_855042_1.stdout parsl.dataflow.dflow INFO: Standard error for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/81ecc660-121d-4c7d-bbbb-ff626d43f903_characterizeImage_855042_1.stderr parsl.dataflow.dflow INFO: Standard output for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/ce71a764-566e-4af7-ac27-bb28363fd409_isr_855042_1.stdout parsl.dataflow.dflow INFO: Standard error for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/ce71a764-566e-4af7-ac27-bb28363fd409_isr_855042_1.stderr parsl.dataflow.dflow INFO: Task 546 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c9d0c0a42440c2619d5a07f54938d9c with result from task 546 parsl.dataflow.dflow DEBUG: Task 546 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 547 has memoization hash 4bbc6e377e9b0a7069c6498046237c8c parsl.dataflow.memoization INFO: Task 547 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa457018a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 547 try 0 launched on executor multi with executor id 522 parsl.dataflow.dflow INFO: Standard output for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/4315ea40-e446-4506-b992-805adc806852_characterizeImage_855138_62.stdout parsl.dataflow.dflow INFO: Standard error for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/4315ea40-e446-4506-b992-805adc806852_characterizeImage_855138_62.stderr parsl.dataflow.dflow INFO: Standard output for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/19c510a4-32c9-4c8e-ba61-e11f846fb755_isr_855138_62.stdout parsl.dataflow.dflow INFO: Standard error for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/19c510a4-32c9-4c8e-ba61-e11f846fb755_isr_855138_62.stderr parsl.dataflow.dflow INFO: Task 567 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 854802138eadbcd548a21c6b87d5f079 with result from task 567 parsl.dataflow.dflow DEBUG: Task 567 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 568 has memoization hash 33cc4bed79fa4a1ac43e1ded7bbb13a4 parsl.dataflow.memoization INFO: Task 568 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa457032e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 568 try 0 launched on executor multi with executor id 523 parsl.dataflow.dflow INFO: Standard output for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/96036120-8b78-4714-9391-8510d32c9df3_characterizeImage_855043_50.stdout parsl.dataflow.dflow INFO: Standard error for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/96036120-8b78-4714-9391-8510d32c9df3_characterizeImage_855043_50.stderr parsl.dataflow.dflow INFO: Standard output for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/6becd73c-abe1-49ea-bd82-13ea2cc8949e_isr_855043_50.stdout parsl.dataflow.dflow INFO: Standard error for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/6becd73c-abe1-49ea-bd82-13ea2cc8949e_isr_855043_50.stderr parsl.dataflow.dflow INFO: Task 561 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79d3926b8903f884f5d762ba1215fa8b with result from task 561 parsl.dataflow.dflow DEBUG: Task 561 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 562 has memoization hash a38c4bb80ffd239860a4eee7fb5ed745 parsl.dataflow.memoization INFO: Task 562 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45702ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 562 try 0 launched on executor multi with executor id 524 parsl.dataflow.dflow INFO: Standard output for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/0ed78163-9a35-478f-9e9b-1efab5d52fe7_characterizeImage_855138_16.stdout parsl.dataflow.dflow INFO: Standard error for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/0ed78163-9a35-478f-9e9b-1efab5d52fe7_characterizeImage_855138_16.stderr parsl.dataflow.dflow INFO: Standard output for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/2bd76537-b61d-47a2-948d-75eadd2a5edc_isr_855138_16.stdout parsl.dataflow.dflow INFO: Standard error for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/2bd76537-b61d-47a2-948d-75eadd2a5edc_isr_855138_16.stderr parsl.dataflow.dflow INFO: Task 552 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0968c759e1b520f2988afa078e1de48 with result from task 552 parsl.dataflow.dflow DEBUG: Task 552 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 553 has memoization hash 4c48847435dd16c7d3961d4f2c2654ff parsl.dataflow.memoization INFO: Task 553 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45702020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 553 try 0 launched on executor multi with executor id 525 parsl.dataflow.dflow INFO: Standard output for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/71f72655-9033-4b00-8b86-2df3538044fe_characterizeImage_855044_14.stdout parsl.dataflow.dflow INFO: Standard error for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/71f72655-9033-4b00-8b86-2df3538044fe_characterizeImage_855044_14.stderr parsl.dataflow.dflow INFO: Standard output for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/59a70cea-9c6b-4522-a3b9-b2ace1bde2c0_isr_855044_14.stdout parsl.dataflow.dflow INFO: Standard error for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/59a70cea-9c6b-4522-a3b9-b2ace1bde2c0_isr_855044_14.stderr parsl.dataflow.dflow INFO: Task 558 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71367250e0e47ff555f5ab1fdabb1896 with result from task 558 parsl.dataflow.dflow DEBUG: Task 558 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 559 has memoization hash 0bc2ecbec6bfee3f800e01c51610afe3 parsl.dataflow.memoization INFO: Task 559 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa457025c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 559 try 0 launched on executor multi with executor id 526 parsl.dataflow.dflow INFO: Standard output for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/1eb8ae48-f248-4f25-9e2e-b35e9097220e_characterizeImage_855138_50.stdout parsl.dataflow.dflow INFO: Standard error for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/1eb8ae48-f248-4f25-9e2e-b35e9097220e_characterizeImage_855138_50.stderr parsl.dataflow.dflow INFO: Standard output for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/ff8e4113-73e2-4dcc-9479-cc2965049555_isr_855138_50.stdout parsl.dataflow.dflow INFO: Standard error for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/ff8e4113-73e2-4dcc-9479-cc2965049555_isr_855138_50.stderr parsl.dataflow.dflow INFO: Task 660 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f38aa51b33785206220f7b3ed2ad1533 with result from task 660 parsl.dataflow.dflow DEBUG: Task 660 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 661 has memoization hash bb9d51823482f02f1d66630f437e1565 parsl.dataflow.memoization INFO: Task 661 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 661 try 0 launched on executor multi with executor id 527 parsl.dataflow.dflow INFO: Standard output for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/5185327c-59b2-4129-9cd2-58854b99ac1c_characterizeImage_855136_47.stdout parsl.dataflow.dflow INFO: Standard error for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/5185327c-59b2-4129-9cd2-58854b99ac1c_characterizeImage_855136_47.stderr parsl.dataflow.dflow INFO: Standard output for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/acfbdb89-4612-41c8-905d-b2c17354061a_isr_855136_47.stdout parsl.dataflow.dflow INFO: Standard error for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/acfbdb89-4612-41c8-905d-b2c17354061a_isr_855136_47.stderr parsl.dataflow.dflow INFO: Task 795 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87f9fe05667b7ad458534905b98dbf30 with result from task 795 parsl.dataflow.dflow DEBUG: Task 795 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 796 has memoization hash 977ee046be8ec24e8fa23f3bb2a4d00c parsl.dataflow.memoization INFO: Task 796 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 796 try 0 launched on executor multi with executor id 528 parsl.dataflow.dflow INFO: Standard output for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/9144c2da-e3b9-44d0-9230-edb4da76e6dd_characterizeImage_855136_39.stdout parsl.dataflow.dflow INFO: Standard error for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/9144c2da-e3b9-44d0-9230-edb4da76e6dd_characterizeImage_855136_39.stderr parsl.dataflow.dflow INFO: Standard output for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/722b50d0-77a2-446a-b260-fd575fa7623b_isr_855136_39.stdout parsl.dataflow.dflow INFO: Standard error for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/722b50d0-77a2-446a-b260-fd575fa7623b_isr_855136_39.stderr parsl.dataflow.dflow INFO: Task 549 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry afa52808ee6372d1f3261c63526e56df with result from task 549 parsl.dataflow.dflow DEBUG: Task 549 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 550 has memoization hash 5e41793b2157dedef250dfd093d9e825 parsl.dataflow.memoization INFO: Task 550 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45701bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 550 try 0 launched on executor multi with executor id 529 parsl.dataflow.dflow INFO: Standard output for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/de577080-318d-4964-bbec-e20e0cd52651_characterizeImage_855044_62.stdout parsl.dataflow.dflow INFO: Standard error for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/de577080-318d-4964-bbec-e20e0cd52651_characterizeImage_855044_62.stderr parsl.dataflow.dflow INFO: Standard output for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/7e1d558a-fdb4-4f85-822e-6335ed8c9f9b_isr_855044_62.stdout parsl.dataflow.dflow INFO: Standard error for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/7e1d558a-fdb4-4f85-822e-6335ed8c9f9b_isr_855044_62.stderr parsl.dataflow.dflow INFO: Task 579 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eed3514a57245b7e3b51bfd5af55ac0b with result from task 579 parsl.dataflow.dflow DEBUG: Task 579 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 580 has memoization hash f951cb7270da796f820346ac089394c8 parsl.dataflow.memoization INFO: Task 580 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa457434c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 580 try 0 launched on executor multi with executor id 530 parsl.dataflow.dflow INFO: Standard output for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/1a51505a-2a77-4d9e-a240-cc02b83866d1_characterizeImage_855043_58.stdout parsl.dataflow.dflow INFO: Standard error for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/1a51505a-2a77-4d9e-a240-cc02b83866d1_characterizeImage_855043_58.stderr parsl.dataflow.dflow INFO: Standard output for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/87ae1638-4224-40da-b09e-8f406807252e_isr_855043_58.stdout parsl.dataflow.dflow INFO: Standard error for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/87ae1638-4224-40da-b09e-8f406807252e_isr_855043_58.stderr parsl.dataflow.dflow INFO: Task 777 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d3f7edb7c0513ace76896c7bbda44c3 with result from task 777 parsl.dataflow.dflow DEBUG: Task 777 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 778 has memoization hash 6b17b9396c5aa649a0ce26487b58ac56 parsl.dataflow.memoization INFO: Task 778 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 778 try 0 launched on executor multi with executor id 531 parsl.dataflow.dflow INFO: Standard output for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/7839a0a9-b0da-4e37-b8f9-eba6dfac75de_characterizeImage_855044_26.stdout parsl.dataflow.dflow INFO: Standard error for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/7839a0a9-b0da-4e37-b8f9-eba6dfac75de_characterizeImage_855044_26.stderr parsl.dataflow.dflow INFO: Standard output for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/31ad4672-437f-4cd2-ad39-41e9c60266ca_isr_855044_26.stdout parsl.dataflow.dflow INFO: Standard error for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/31ad4672-437f-4cd2-ad39-41e9c60266ca_isr_855044_26.stderr parsl.dataflow.dflow INFO: Task 783 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 34f69d6d709ca05a7a49258f22b9c374 with result from task 783 parsl.dataflow.dflow DEBUG: Task 783 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 784 has memoization hash 7dd76d711b92f7b636dabfdc278b205e parsl.dataflow.memoization INFO: Task 784 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 784 try 0 launched on executor multi with executor id 532 parsl.dataflow.dflow INFO: Standard output for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/74d32e1e-6641-482d-a536-d1178682d1a8_characterizeImage_855043_57.stdout parsl.dataflow.dflow INFO: Standard error for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/74d32e1e-6641-482d-a536-d1178682d1a8_characterizeImage_855043_57.stderr parsl.dataflow.dflow INFO: Standard output for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/11023720-c556-4269-8d7f-0f9d1e523a3a_isr_855043_57.stdout parsl.dataflow.dflow INFO: Standard error for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/11023720-c556-4269-8d7f-0f9d1e523a3a_isr_855043_57.stderr parsl.dataflow.dflow INFO: Task 789 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d32f27383a5fa94ca6daf2fddd6d3d6 with result from task 789 parsl.dataflow.dflow DEBUG: Task 789 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 790 has memoization hash 5c10b95635a11785a695ebe4b8b8e88d parsl.dataflow.memoization INFO: Task 790 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 790 try 0 launched on executor multi with executor id 533 parsl.dataflow.dflow INFO: Standard output for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/fb4cfe25-d82e-49df-a6f9-207be186bc25_characterizeImage_855044_35.stdout parsl.dataflow.dflow INFO: Standard error for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/fb4cfe25-d82e-49df-a6f9-207be186bc25_characterizeImage_855044_35.stderr parsl.dataflow.dflow INFO: Standard output for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/882819b5-6792-469f-9a9f-e3d8878d3080_isr_855044_35.stdout parsl.dataflow.dflow INFO: Standard error for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/882819b5-6792-469f-9a9f-e3d8878d3080_isr_855044_35.stderr parsl.dataflow.dflow INFO: Task 594 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73655b27bd1312cb08aa01e6acaeab22 with result from task 594 parsl.dataflow.dflow DEBUG: Task 594 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 595 has memoization hash 6a5be1ea6724fa9b116cd160e08e79b3 parsl.dataflow.memoization INFO: Task 595 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45740860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 595 try 0 launched on executor multi with executor id 534 parsl.dataflow.dflow INFO: Standard output for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/69f251fe-c21e-46fd-adef-dd0cc1e49f52_characterizeImage_855043_2.stdout parsl.dataflow.dflow INFO: Standard error for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/69f251fe-c21e-46fd-adef-dd0cc1e49f52_characterizeImage_855043_2.stderr parsl.dataflow.dflow INFO: Standard output for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/93c62f2c-782f-4cb7-a9e1-fd456f85e653_isr_855043_2.stdout parsl.dataflow.dflow INFO: Standard error for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/93c62f2c-782f-4cb7-a9e1-fd456f85e653_isr_855043_2.stderr parsl.dataflow.dflow INFO: Task 657 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d75495a96f491df760efdb8fb622b458 with result from task 657 parsl.dataflow.dflow DEBUG: Task 657 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 658 has memoization hash 7363a7271252e9e77b0f179eeaaaddb8 parsl.dataflow.memoization INFO: Task 658 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 658 try 0 launched on executor multi with executor id 535 parsl.dataflow.dflow INFO: Standard output for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/4cdb4b4a-5888-42f3-a19b-ca8672765ff7_characterizeImage_855042_36.stdout parsl.dataflow.dflow INFO: Standard error for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/4cdb4b4a-5888-42f3-a19b-ca8672765ff7_characterizeImage_855042_36.stderr parsl.dataflow.dflow INFO: Standard output for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/6cc02eb7-13c2-4813-9ee4-30316f3c73ce_isr_855042_36.stdout parsl.dataflow.dflow INFO: Standard error for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/6cc02eb7-13c2-4813-9ee4-30316f3c73ce_isr_855042_36.stderr parsl.dataflow.dflow INFO: Task 882 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f835d2c0cc76cb9a5db89377e4ece3e with result from task 882 parsl.dataflow.dflow DEBUG: Task 882 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 883 has memoization hash b973a94ef7b766f9cbaa1562ff69e0b3 parsl.dataflow.memoization INFO: Task 883 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45667920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 883 try 0 launched on executor multi with executor id 536 parsl.dataflow.dflow INFO: Standard output for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/daaf5bb5-4445-4596-8c3a-1c59b07acc2f_characterizeImage_855138_3.stdout parsl.dataflow.dflow INFO: Standard error for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/daaf5bb5-4445-4596-8c3a-1c59b07acc2f_characterizeImage_855138_3.stderr parsl.dataflow.dflow INFO: Standard output for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/d0d7963a-91ce-453c-9381-d623d388ac22_isr_855138_3.stdout parsl.dataflow.dflow INFO: Standard error for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/d0d7963a-91ce-453c-9381-d623d388ac22_isr_855138_3.stderr parsl.dataflow.dflow INFO: Task 792 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d95bb808aa5641e716f3018e14c4a6c8 with result from task 792 parsl.dataflow.dflow DEBUG: Task 792 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 793 has memoization hash 340a0cb1a57163152c5b89daee2456bb parsl.dataflow.memoization INFO: Task 793 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 793 try 0 launched on executor multi with executor id 537 parsl.dataflow.dflow INFO: Standard output for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/da3b2a11-677c-441d-b317-67909a54f1de_characterizeImage_855043_49.stdout parsl.dataflow.dflow INFO: Standard error for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/da3b2a11-677c-441d-b317-67909a54f1de_characterizeImage_855043_49.stderr parsl.dataflow.dflow INFO: Standard output for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/4119a780-6cf2-4e1a-9d28-fdec4737bcc7_isr_855043_49.stdout parsl.dataflow.dflow INFO: Standard error for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/4119a780-6cf2-4e1a-9d28-fdec4737bcc7_isr_855043_49.stderr parsl.dataflow.dflow INFO: Task 774 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 318b107635ab1287259e28eef7ca853e with result from task 774 parsl.dataflow.dflow DEBUG: Task 774 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 775 has memoization hash c7d557705dc70e9273852840735d842c parsl.dataflow.memoization INFO: Task 775 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d3ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 775 try 0 launched on executor multi with executor id 538 parsl.dataflow.dflow INFO: Standard output for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/143b6809-5c04-4ed0-8eca-437097873506_characterizeImage_855138_38.stdout parsl.dataflow.dflow INFO: Standard error for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/143b6809-5c04-4ed0-8eca-437097873506_characterizeImage_855138_38.stderr parsl.dataflow.dflow INFO: Standard output for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/cd2d9445-5c55-467b-a5a3-d860c2a45957_isr_855138_38.stdout parsl.dataflow.dflow INFO: Standard error for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/cd2d9445-5c55-467b-a5a3-d860c2a45957_isr_855138_38.stderr parsl.dataflow.dflow INFO: Task 780 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23194d8c567897bf0b63cf0a9ea5bf30 with result from task 780 parsl.dataflow.dflow DEBUG: Task 780 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 781 has memoization hash be255e7f575655d67c58982ac04175e6 parsl.dataflow.memoization INFO: Task 781 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 781 try 0 launched on executor multi with executor id 539 parsl.dataflow.dflow INFO: Standard output for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/9aff9434-8a09-4f9d-9841-4e024676275f_characterizeImage_855042_22.stdout parsl.dataflow.dflow INFO: Standard error for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/9aff9434-8a09-4f9d-9841-4e024676275f_characterizeImage_855042_22.stderr parsl.dataflow.dflow INFO: Standard output for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/f84ac8cb-acf5-4fe6-81ef-e5962ced860a_isr_855042_22.stdout parsl.dataflow.dflow INFO: Standard error for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/f84ac8cb-acf5-4fe6-81ef-e5962ced860a_isr_855042_22.stderr parsl.dataflow.dflow INFO: Task 564 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3649ec3db779fb2c0592ae892e3a39ad with result from task 564 parsl.dataflow.dflow DEBUG: Task 564 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 565 has memoization hash c143306785900dbb7a1e3c7716bfec0a parsl.dataflow.memoization INFO: Task 565 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45702d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 565 try 0 launched on executor multi with executor id 540 parsl.dataflow.dflow INFO: Standard output for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/c0895e9e-fd27-4c41-a41b-c6ea86b0efa5_characterizeImage_855136_7.stdout parsl.dataflow.dflow INFO: Standard error for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/c0895e9e-fd27-4c41-a41b-c6ea86b0efa5_characterizeImage_855136_7.stderr parsl.dataflow.dflow INFO: Standard output for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/0a8fdfdb-c28a-4cc4-9c90-866c5a9795e6_isr_855136_7.stdout parsl.dataflow.dflow INFO: Standard error for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/0a8fdfdb-c28a-4cc4-9c90-866c5a9795e6_isr_855136_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:17 proc_lsst.multi:146 [INFO] found job 16912303 in provider astro proc_lsst.multi INFO: found job 16912303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 771 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ff8a041501a2c99094968c7fe007409 with result from task 771 parsl.dataflow.dflow DEBUG: Task 771 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 772 has memoization hash f761cef53898b1e8f335be29d3d3741a parsl.dataflow.memoization INFO: Task 772 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d3f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 772 try 0 launched on executor multi with executor id 541 parsl.dataflow.dflow INFO: Standard output for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/6da68ef4-c372-4ca2-81a0-8df5a06b9fd0_characterizeImage_855043_37.stdout parsl.dataflow.dflow INFO: Standard error for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/6da68ef4-c372-4ca2-81a0-8df5a06b9fd0_characterizeImage_855043_37.stderr parsl.dataflow.dflow INFO: Standard output for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/8e69ed9a-5228-4370-a9e6-67dc732f69ba_isr_855043_37.stdout parsl.dataflow.dflow INFO: Standard error for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/8e69ed9a-5228-4370-a9e6-67dc732f69ba_isr_855043_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:18 proc_lsst.multi:146 [INFO] found job 16912304 in provider astro proc_lsst.multi INFO: found job 16912304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 837 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 402c36aaa254565403a076e792eac65c with result from task 837 parsl.dataflow.dflow DEBUG: Task 837 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 838 has memoization hash 6951065dc87249f7f05e141d1eabd822 parsl.dataflow.memoization INFO: Task 838 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d2200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 838 try 0 launched on executor multi with executor id 542 parsl.dataflow.dflow INFO: Standard output for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/46441930-63f7-430b-86f1-53c061a311e1_characterizeImage_855136_9.stdout parsl.dataflow.dflow INFO: Standard error for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/46441930-63f7-430b-86f1-53c061a311e1_characterizeImage_855136_9.stderr parsl.dataflow.dflow INFO: Standard output for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/e9fee100-7b4d-42db-9652-3b03eee83ecd_isr_855136_9.stdout parsl.dataflow.dflow INFO: Standard error for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/e9fee100-7b4d-42db-9652-3b03eee83ecd_isr_855136_9.stderr parsl.dataflow.dflow INFO: Task 786 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf4c62e494e317df4a1b3f7cbaba6d77 with result from task 786 parsl.dataflow.dflow DEBUG: Task 786 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 787 has memoization hash b15f2c4c8931e897ce7eef958be8dd23 parsl.dataflow.memoization INFO: Task 787 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 787 try 0 launched on executor multi with executor id 543 parsl.dataflow.dflow INFO: Standard output for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/55393c2a-c811-47c0-a177-2a393a0b29f6_characterizeImage_855137_50.stdout parsl.dataflow.dflow INFO: Standard error for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/55393c2a-c811-47c0-a177-2a393a0b29f6_characterizeImage_855137_50.stderr parsl.dataflow.dflow INFO: Standard output for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/4a85b5ef-9c6d-4dc2-bbf0-23c034f1e76b_isr_855137_50.stdout parsl.dataflow.dflow INFO: Standard error for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/4a85b5ef-9c6d-4dc2-bbf0-23c034f1e76b_isr_855137_50.stderr parsl.dataflow.dflow INFO: Task 597 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8037ef1571bc80df2e58ed32ff134cc5 with result from task 597 parsl.dataflow.dflow DEBUG: Task 597 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 598 has memoization hash f728d28cbdc9b5b37ff2a55a19e980ff parsl.dataflow.memoization INFO: Task 598 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45740a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 598 try 0 launched on executor multi with executor id 544 parsl.dataflow.dflow INFO: Standard output for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/6cacb009-b5f7-47f6-92fd-064222895fec_characterizeImage_855042_33.stdout parsl.dataflow.dflow INFO: Standard error for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/6cacb009-b5f7-47f6-92fd-064222895fec_characterizeImage_855042_33.stderr parsl.dataflow.dflow INFO: Standard output for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/5d9ef8b4-16fe-4181-b436-37fc2b0f9812_isr_855042_33.stdout parsl.dataflow.dflow INFO: Standard error for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/5d9ef8b4-16fe-4181-b436-37fc2b0f9812_isr_855042_33.stderr parsl.dataflow.dflow INFO: Task 819 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 143f20f71f3d40b916f9f656801d50fe with result from task 819 parsl.dataflow.dflow DEBUG: Task 819 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 820 has memoization hash 7bba88759cfc233438c595769bbad1ed parsl.dataflow.memoization INFO: Task 820 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 820 try 0 launched on executor multi with executor id 545 parsl.dataflow.dflow INFO: Standard output for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/75b934e6-c91a-42e4-8d3a-502a30b56220_characterizeImage_855042_58.stdout parsl.dataflow.dflow INFO: Standard error for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/75b934e6-c91a-42e4-8d3a-502a30b56220_characterizeImage_855042_58.stderr parsl.dataflow.dflow INFO: Standard output for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/5567f9c2-5569-4a0f-b82d-c5ba949b953a_isr_855042_58.stdout parsl.dataflow.dflow INFO: Standard error for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/5567f9c2-5569-4a0f-b82d-c5ba949b953a_isr_855042_58.stderr parsl.dataflow.dflow INFO: Task 891 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96bdd4417c529a8cae27808193209bbc with result from task 891 parsl.dataflow.dflow DEBUG: Task 891 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 892 has memoization hash 8e09ed84ee829845fd8ff196cc050da1 parsl.dataflow.memoization INFO: Task 892 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45498b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 892 try 0 launched on executor multi with executor id 546 parsl.dataflow.dflow INFO: Standard output for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/60412ab4-d2b1-4673-9626-0a9697d8aae3_characterizeImage_855138_40.stdout parsl.dataflow.dflow INFO: Standard error for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/60412ab4-d2b1-4673-9626-0a9697d8aae3_characterizeImage_855138_40.stderr parsl.dataflow.dflow INFO: Standard output for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/71a522e0-f337-47d7-ac15-eaefe701b767_isr_855138_40.stdout parsl.dataflow.dflow INFO: Standard error for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/71a522e0-f337-47d7-ac15-eaefe701b767_isr_855138_40.stderr parsl.dataflow.dflow INFO: Task 873 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91f6627379513df09e6de29e418cf7da with result from task 873 parsl.dataflow.dflow DEBUG: Task 873 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 874 has memoization hash d5ee3c2628c49d0e826ae7f7da60ef7f parsl.dataflow.memoization INFO: Task 874 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45667100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 874 try 0 launched on executor multi with executor id 547 parsl.dataflow.dflow INFO: Standard output for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/ec5235e9-6c9e-47d1-babb-4ec2b9224d0f_characterizeImage_855136_2.stdout parsl.dataflow.dflow INFO: Standard error for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/ec5235e9-6c9e-47d1-babb-4ec2b9224d0f_characterizeImage_855136_2.stderr parsl.dataflow.dflow INFO: Standard output for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/7e4d5ace-abc8-4172-9948-f20b706a5277_isr_855136_2.stdout parsl.dataflow.dflow INFO: Standard error for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/7e4d5ace-abc8-4172-9948-f20b706a5277_isr_855136_2.stderr parsl.dataflow.dflow INFO: Task 825 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 26b4698220c33dc4ae36026eb01708a2 with result from task 825 parsl.dataflow.dflow DEBUG: Task 825 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 826 has memoization hash 3b6289d5bbc18069b0c9a400278b721e parsl.dataflow.memoization INFO: Task 826 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456647c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 826 try 0 launched on executor multi with executor id 548 parsl.dataflow.dflow INFO: Standard output for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/36048fa2-8461-4d1a-b30b-664b1ea3e852_characterizeImage_855044_61.stdout parsl.dataflow.dflow INFO: Standard error for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/36048fa2-8461-4d1a-b30b-664b1ea3e852_characterizeImage_855044_61.stderr parsl.dataflow.dflow INFO: Standard output for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/3f8c4794-6f66-4461-b4d0-b89a94902aba_isr_855044_61.stdout parsl.dataflow.dflow INFO: Standard error for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/3f8c4794-6f66-4461-b4d0-b89a94902aba_isr_855044_61.stderr parsl.dataflow.dflow INFO: Task 885 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc376446125c0945b9e0b34937027d35 with result from task 885 parsl.dataflow.dflow DEBUG: Task 885 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 886 has memoization hash 6d47683908acf87f75a95b9cee390ead parsl.dataflow.memoization INFO: Task 886 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45667d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 886 try 0 launched on executor multi with executor id 549 parsl.dataflow.dflow INFO: Standard output for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/993efa91-b6f2-40b5-9730-6867d85689c3_characterizeImage_855042_53.stdout parsl.dataflow.dflow INFO: Standard error for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/993efa91-b6f2-40b5-9730-6867d85689c3_characterizeImage_855042_53.stderr parsl.dataflow.dflow INFO: Standard output for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/d00f4cea-261f-485a-b366-f91e72175331_isr_855042_53.stdout parsl.dataflow.dflow INFO: Standard error for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/d00f4cea-261f-485a-b366-f91e72175331_isr_855042_53.stderr parsl.dataflow.dflow INFO: Task 606 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ff85eb99075a3ab28714a1e2397f4d6 with result from task 606 parsl.dataflow.dflow DEBUG: Task 606 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 607 has memoization hash 7a8a73f385c7c808b7183d1ac78cb328 parsl.dataflow.memoization INFO: Task 607 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa457414e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 607 try 0 launched on executor multi with executor id 550 parsl.dataflow.dflow INFO: Standard output for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/7ecebbf8-697e-4c14-acfe-05eb3a13e303_characterizeImage_855138_5.stdout parsl.dataflow.dflow INFO: Standard error for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/7ecebbf8-697e-4c14-acfe-05eb3a13e303_characterizeImage_855138_5.stderr parsl.dataflow.dflow INFO: Standard output for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/a5b43f58-d427-49ca-bba3-878b52505dcf_isr_855138_5.stdout parsl.dataflow.dflow INFO: Standard error for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/a5b43f58-d427-49ca-bba3-878b52505dcf_isr_855138_5.stderr parsl.dataflow.dflow INFO: Task 603 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10a43a1a84d65dfd80214be8df03bcc1 with result from task 603 parsl.dataflow.dflow DEBUG: Task 603 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 604 has memoization hash 7c80611ddf11d35d8d727ea025bb9136 parsl.dataflow.memoization INFO: Task 604 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45741260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 604 try 0 launched on executor multi with executor id 551 parsl.dataflow.dflow INFO: Standard output for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/2bcbb6e2-c6c2-43c1-b057-c8834197bea0_characterizeImage_855137_23.stdout parsl.dataflow.dflow INFO: Standard error for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/2bcbb6e2-c6c2-43c1-b057-c8834197bea0_characterizeImage_855137_23.stderr parsl.dataflow.dflow INFO: Standard output for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/4c455147-56b0-4fd6-b83a-75276741d57f_isr_855137_23.stdout parsl.dataflow.dflow INFO: Standard error for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/4c455147-56b0-4fd6-b83a-75276741d57f_isr_855137_23.stderr parsl.dataflow.dflow INFO: Task 609 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7098f097c14a83b9e0bc0b26f6f212b with result from task 609 parsl.dataflow.dflow DEBUG: Task 609 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 610 has memoization hash 91fadd832349dff6f5843dde51dc4995 parsl.dataflow.memoization INFO: Task 610 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45741800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 610 try 0 launched on executor multi with executor id 552 parsl.dataflow.dflow INFO: Standard output for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/eecd9692-8df6-4407-a0ae-d5cd324a5cff_characterizeImage_855044_46.stdout parsl.dataflow.dflow INFO: Standard error for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/eecd9692-8df6-4407-a0ae-d5cd324a5cff_characterizeImage_855044_46.stderr parsl.dataflow.dflow INFO: Standard output for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/dc5c83d1-9067-4b29-8fba-e11209a572cc_isr_855044_46.stdout parsl.dataflow.dflow INFO: Standard error for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/dc5c83d1-9067-4b29-8fba-e11209a572cc_isr_855044_46.stderr parsl.dataflow.dflow INFO: Task 600 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd9b146dd8c9b3290f4618e331e0a1b1 with result from task 600 parsl.dataflow.dflow DEBUG: Task 600 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 601 has memoization hash e660339cbdae1d2a0881e781075fba01 parsl.dataflow.memoization INFO: Task 601 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45740f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 601 try 0 launched on executor multi with executor id 553 parsl.dataflow.dflow INFO: Standard output for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/97404417-791e-4ba5-83a1-8532ef9938dd_characterizeImage_855136_23.stdout parsl.dataflow.dflow INFO: Standard error for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/97404417-791e-4ba5-83a1-8532ef9938dd_characterizeImage_855136_23.stderr parsl.dataflow.dflow INFO: Standard output for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/9bce308c-adc0-4c79-8b96-ee11f06fa84b_isr_855136_23.stdout parsl.dataflow.dflow INFO: Standard error for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/9bce308c-adc0-4c79-8b96-ee11f06fa84b_isr_855136_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:20 proc_lsst.multi:146 [INFO] found job 16912305 in provider astro proc_lsst.multi INFO: found job 16912305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 915 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6ea43aa2b4dd7159693ba7a33d7251a with result from task 915 parsl.dataflow.dflow DEBUG: Task 915 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 916 has memoization hash 833987b0c2882c73644cf35d553fb6cd parsl.dataflow.memoization INFO: Task 916 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4549a5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 916 try 0 launched on executor multi with executor id 554 parsl.dataflow.dflow INFO: Standard output for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/1ca73511-5813-4310-a233-081941d8b622_characterizeImage_855136_57.stdout parsl.dataflow.dflow INFO: Standard error for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/1ca73511-5813-4310-a233-081941d8b622_characterizeImage_855136_57.stderr parsl.dataflow.dflow INFO: Standard output for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/a9aadfea-2213-4059-bc4b-ce9054709382_isr_855136_57.stdout parsl.dataflow.dflow INFO: Standard error for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/a9aadfea-2213-4059-bc4b-ce9054709382_isr_855136_57.stderr parsl.dataflow.dflow INFO: Task 390 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 88155f41c1482dbe8921c3beeb9458a0 with result from task 390 parsl.dataflow.dflow DEBUG: Task 390 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 391 has memoization hash a97b8755c078a2e49ccc0564639c1bb4 parsl.dataflow.memoization INFO: Task 391 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4687bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 391 try 0 launched on executor multi with executor id 555 parsl.dataflow.dflow INFO: Standard output for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/d7f79a9c-f7aa-42a3-84c1-ce23407ffc8d_characterizeImage_855137_2.stdout parsl.dataflow.dflow INFO: Standard error for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/d7f79a9c-f7aa-42a3-84c1-ce23407ffc8d_characterizeImage_855137_2.stderr parsl.dataflow.dflow INFO: Standard output for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/5250b605-9001-45d2-94e9-5126ee258c3a_isr_855137_2.stdout parsl.dataflow.dflow INFO: Standard error for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/5250b605-9001-45d2-94e9-5126ee258c3a_isr_855137_2.stderr parsl.dataflow.dflow INFO: Task 615 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3cde280be35757b85eb02f5cd4864edd with result from task 615 parsl.dataflow.dflow DEBUG: Task 615 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 616 has memoization hash 773bdedd20116b56fb68dd40679d8489 parsl.dataflow.memoization INFO: Task 616 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45741ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 616 try 0 launched on executor multi with executor id 556 parsl.dataflow.dflow INFO: Standard output for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/ea6161f5-ed23-4a67-8519-3b7e412517f7_characterizeImage_855136_49.stdout parsl.dataflow.dflow INFO: Standard error for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/ea6161f5-ed23-4a67-8519-3b7e412517f7_characterizeImage_855136_49.stderr parsl.dataflow.dflow INFO: Standard output for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/9bbe0a88-c4a8-42bc-bc61-78681af41e45_isr_855136_49.stdout parsl.dataflow.dflow INFO: Standard error for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/9bbe0a88-c4a8-42bc-bc61-78681af41e45_isr_855136_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:20 proc_lsst.multi:146 [INFO] found job 16912306 in provider astro proc_lsst.multi INFO: found job 16912306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 903 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4c0d5d9388603ab8993868653e38fd7 with result from task 903 parsl.dataflow.dflow DEBUG: Task 903 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 904 has memoization hash c40e3de881b69a96ee6846378b43b902 parsl.dataflow.memoization INFO: Task 904 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45499940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 904 try 0 launched on executor multi with executor id 557 parsl.dataflow.dflow INFO: Standard output for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/7fd5cf92-1dbe-4bbc-b1b7-770ace5ff059_characterizeImage_855043_56.stdout parsl.dataflow.dflow INFO: Standard error for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/7fd5cf92-1dbe-4bbc-b1b7-770ace5ff059_characterizeImage_855043_56.stderr parsl.dataflow.dflow INFO: Standard output for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/633568b0-a778-416a-826b-7c4d54a70afe_isr_855043_56.stdout parsl.dataflow.dflow INFO: Standard error for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/633568b0-a778-416a-826b-7c4d54a70afe_isr_855043_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:21 proc_lsst.multi:146 [INFO] found job 16912307 in provider astro proc_lsst.multi INFO: found job 16912307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 906 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 874f50b6e0e3dd71bd27e3addb540ce5 with result from task 906 parsl.dataflow.dflow DEBUG: Task 906 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 907 has memoization hash 7767485ceb1823884cfc2dd4fb36c8e0 parsl.dataflow.memoization INFO: Task 907 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45499c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 907 try 0 launched on executor multi with executor id 558 parsl.dataflow.dflow INFO: Standard output for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/c137c492-7af8-47ab-a94c-4743e0c2fa53_characterizeImage_855042_8.stdout parsl.dataflow.dflow INFO: Standard error for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/c137c492-7af8-47ab-a94c-4743e0c2fa53_characterizeImage_855042_8.stderr parsl.dataflow.dflow INFO: Standard output for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/4f01a72b-df1c-41f1-becd-7b90de6136c9_isr_855042_8.stdout parsl.dataflow.dflow INFO: Standard error for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/4f01a72b-df1c-41f1-becd-7b90de6136c9_isr_855042_8.stderr parsl.dataflow.dflow INFO: Task 336 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 154d55cbf959fecb7d299785ec11041a with result from task 336 parsl.dataflow.dflow DEBUG: Task 336 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 337 has memoization hash 86e7412b47ae07a4f9c50c4b0f5eb658 parsl.dataflow.memoization INFO: Task 337 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa468784a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 337 try 0 launched on executor multi with executor id 559 parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/071f77e6-ab22-4547-9959-6c4d543c20d5_characterizeImage_855043_5.stdout parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/071f77e6-ab22-4547-9959-6c4d543c20d5_characterizeImage_855043_5.stderr parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/5def27a8-b097-40bc-9bbf-5a078a34e457_isr_855043_5.stdout parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/5def27a8-b097-40bc-9bbf-5a078a34e457_isr_855043_5.stderr parsl.dataflow.dflow INFO: Task 321 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b8df6ab361dc027a924b6a59f44e81f with result from task 321 parsl.dataflow.dflow DEBUG: Task 321 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 322 has memoization hash 0c98dd3f8e2816187e679f10c2911d0d parsl.dataflow.memoization INFO: Task 322 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 322 try 0 launched on executor multi with executor id 560 parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/296c1234-f66c-4750-a817-eb5c3b24d7da_characterizeImage_855042_28.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/296c1234-f66c-4750-a817-eb5c3b24d7da_characterizeImage_855042_28.stderr parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/63f481db-e270-47c3-a287-98fc9d25440b_isr_855042_28.stdout parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/63f481db-e270-47c3-a287-98fc9d25440b_isr_855042_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:21 proc_lsst.multi:146 [INFO] found job 16912308 in provider astro proc_lsst.multi INFO: found job 16912308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 858 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2dade082ec4e7888065f28d5e3b7fc94 with result from task 858 parsl.dataflow.dflow DEBUG: Task 858 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 859 has memoization hash b5fdd3b444b10e4fef8c89586befadf8 parsl.dataflow.memoization INFO: Task 859 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454993a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 859 try 0 launched on executor multi with executor id 561 parsl.dataflow.dflow INFO: Standard output for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/2d8f5ec9-35d6-4b0a-9e7d-b4575868a39e_characterizeImage_855136_32.stdout parsl.dataflow.dflow INFO: Standard error for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/2d8f5ec9-35d6-4b0a-9e7d-b4575868a39e_characterizeImage_855136_32.stderr parsl.dataflow.dflow INFO: Standard output for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/cbc1c5ca-cde9-4e4c-b81c-87e92def137f_isr_855136_32.stdout parsl.dataflow.dflow INFO: Standard error for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/cbc1c5ca-cde9-4e4c-b81c-87e92def137f_isr_855136_32.stderr parsl.dataflow.dflow INFO: Task 612 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 610488f08e516a17e312499fd33dd092 with result from task 612 parsl.dataflow.dflow DEBUG: Task 612 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 613 has memoization hash 19cc8ca0c6556a39ee1a7a8d073b6be8 parsl.dataflow.memoization INFO: Task 613 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45741bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 613 try 0 launched on executor multi with executor id 562 parsl.dataflow.dflow INFO: Standard output for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/bdd7c8a3-e23f-41f3-a619-b4b50785512f_characterizeImage_855044_29.stdout parsl.dataflow.dflow INFO: Standard error for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/bdd7c8a3-e23f-41f3-a619-b4b50785512f_characterizeImage_855044_29.stderr parsl.dataflow.dflow INFO: Standard output for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/dc1bb0d5-3398-49d4-8e5a-3c0722fe0575_isr_855044_29.stdout parsl.dataflow.dflow INFO: Standard error for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/dc1bb0d5-3398-49d4-8e5a-3c0722fe0575_isr_855044_29.stderr parsl.dataflow.dflow INFO: Task 843 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c91858a65e5c1482545bd457eb634fa6 with result from task 843 parsl.dataflow.dflow DEBUG: Task 843 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 844 has memoization hash 72f51e751f2db88bb6de80a75c703dd8 parsl.dataflow.memoization INFO: Task 844 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456658a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 844 try 0 launched on executor multi with executor id 563 parsl.dataflow.dflow INFO: Standard output for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/e4b0b5da-f184-4492-bd2e-212e5511bbda_characterizeImage_855043_40.stdout parsl.dataflow.dflow INFO: Standard error for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/e4b0b5da-f184-4492-bd2e-212e5511bbda_characterizeImage_855043_40.stderr parsl.dataflow.dflow INFO: Standard output for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/aa0a4608-0bdd-44e0-a8e0-9ef5915216ba_isr_855043_40.stdout parsl.dataflow.dflow INFO: Standard error for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/aa0a4608-0bdd-44e0-a8e0-9ef5915216ba_isr_855043_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:22 proc_lsst.multi:146 [INFO] found job 16912309 in provider astro proc_lsst.multi INFO: found job 16912309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 876 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a52fd02b7f4571814f891fd25fcd34cc with result from task 876 parsl.dataflow.dflow DEBUG: Task 876 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 877 has memoization hash d89f65c149ec66da160bc7b73ccfe6ef parsl.dataflow.memoization INFO: Task 877 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456672e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 877 try 0 launched on executor multi with executor id 564 parsl.dataflow.dflow INFO: Standard output for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/a6bee093-3a37-4f81-b268-886aa0d97658_characterizeImage_855137_40.stdout parsl.dataflow.dflow INFO: Standard error for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/a6bee093-3a37-4f81-b268-886aa0d97658_characterizeImage_855137_40.stderr parsl.dataflow.dflow INFO: Standard output for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/2783cd13-a407-4283-8c1a-3636ce90da0d_isr_855137_40.stdout parsl.dataflow.dflow INFO: Standard error for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/2783cd13-a407-4283-8c1a-3636ce90da0d_isr_855137_40.stderr parsl.dataflow.dflow INFO: Task 870 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d1d29eb43e10b99d7011c874e437cba with result from task 870 parsl.dataflow.dflow DEBUG: Task 870 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 871 has memoization hash 1132960f11df702ece8695ea5929157b parsl.dataflow.memoization INFO: Task 871 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45666de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 871 try 0 launched on executor multi with executor id 565 parsl.dataflow.dflow INFO: Standard output for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/2f6491fe-d2e3-48d1-b91a-1f4b39a87e2d_characterizeImage_855043_20.stdout parsl.dataflow.dflow INFO: Standard error for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/2f6491fe-d2e3-48d1-b91a-1f4b39a87e2d_characterizeImage_855043_20.stderr parsl.dataflow.dflow INFO: Standard output for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/0a0c6b1d-258a-4722-a8a0-61dd20994512_isr_855043_20.stdout parsl.dataflow.dflow INFO: Standard error for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/0a0c6b1d-258a-4722-a8a0-61dd20994512_isr_855043_20.stderr parsl.dataflow.dflow INFO: Task 393 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb98bde4fa401f995efe052cefe9b677 with result from task 393 parsl.dataflow.dflow DEBUG: Task 393 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 394 has memoization hash f873ea2682fab05347002136b08d470d parsl.dataflow.memoization INFO: Task 394 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4687bec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 394 try 0 launched on executor multi with executor id 566 parsl.dataflow.dflow INFO: Standard output for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/09ece147-f068-49e5-8b33-0a6860c9e8e9_characterizeImage_855044_5.stdout parsl.dataflow.dflow INFO: Standard error for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/09ece147-f068-49e5-8b33-0a6860c9e8e9_characterizeImage_855044_5.stderr parsl.dataflow.dflow INFO: Standard output for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/25e33014-2875-421a-9aac-71492f9e8a14_isr_855044_5.stdout parsl.dataflow.dflow INFO: Standard error for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/25e33014-2875-421a-9aac-71492f9e8a14_isr_855044_5.stderr parsl.dataflow.dflow INFO: Task 816 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50452e2a24619a7264d77abf1ad0ca62 with result from task 816 parsl.dataflow.dflow DEBUG: Task 816 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 817 has memoization hash baad0723413510926538a4a021c5cf9d parsl.dataflow.memoization INFO: Task 817 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 817 try 0 launched on executor multi with executor id 567 parsl.dataflow.dflow INFO: Standard output for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/ff5738e6-f607-4845-a024-30fee2397171_characterizeImage_855043_35.stdout parsl.dataflow.dflow INFO: Standard error for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/ff5738e6-f607-4845-a024-30fee2397171_characterizeImage_855043_35.stderr parsl.dataflow.dflow INFO: Standard output for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/a9ae5fca-f851-49ce-885a-f2107c5cb72e_isr_855043_35.stdout parsl.dataflow.dflow INFO: Standard error for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/a9ae5fca-f851-49ce-885a-f2107c5cb72e_isr_855043_35.stderr parsl.dataflow.dflow INFO: Task 813 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b2f7adf347f358dc10e2d567210e95c8 with result from task 813 parsl.dataflow.dflow DEBUG: Task 813 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 814 has memoization hash 550cd89d33aa0aaa3845dcbb329e74e0 parsl.dataflow.memoization INFO: Task 814 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 814 try 0 launched on executor multi with executor id 568 parsl.dataflow.dflow INFO: Standard output for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/2d02d6b8-8e86-44dc-845b-25351ab8f58e_characterizeImage_855043_31.stdout parsl.dataflow.dflow INFO: Standard error for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/2d02d6b8-8e86-44dc-845b-25351ab8f58e_characterizeImage_855043_31.stderr parsl.dataflow.dflow INFO: Standard output for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/6e645462-531c-4d78-94b8-8149782b61e3_isr_855043_31.stdout parsl.dataflow.dflow INFO: Standard error for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/6e645462-531c-4d78-94b8-8149782b61e3_isr_855043_31.stderr parsl.dataflow.dflow INFO: Task 918 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39b3061e6aec7d4d6a08d84b158bf907 with result from task 918 parsl.dataflow.dflow DEBUG: Task 918 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 919 has memoization hash 11b95bba4e0daed4db9efaf1de5e401c parsl.dataflow.memoization INFO: Task 919 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4549a8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 919 try 0 launched on executor multi with executor id 569 parsl.dataflow.dflow INFO: Standard output for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/80704541-bab4-4b60-9c40-45116b4567c1_characterizeImage_855044_2.stdout parsl.dataflow.dflow INFO: Standard error for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/80704541-bab4-4b60-9c40-45116b4567c1_characterizeImage_855044_2.stderr parsl.dataflow.dflow INFO: Standard output for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/fd7e5c63-97d5-4b6b-942b-194d478822e1_isr_855044_2.stdout parsl.dataflow.dflow INFO: Standard error for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/fd7e5c63-97d5-4b6b-942b-194d478822e1_isr_855044_2.stderr parsl.dataflow.dflow INFO: Task 888 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0961c5493f00c7f0579e64d893839a7f with result from task 888 parsl.dataflow.dflow DEBUG: Task 888 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 889 has memoization hash b3a1ead4a703b60c36d862e21c472d90 parsl.dataflow.memoization INFO: Task 889 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45667c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 889 try 0 launched on executor multi with executor id 570 parsl.dataflow.dflow INFO: Standard output for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/3158853d-2c8e-4d8e-860e-b18c3b2baf82_characterizeImage_855043_52.stdout parsl.dataflow.dflow INFO: Standard error for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/3158853d-2c8e-4d8e-860e-b18c3b2baf82_characterizeImage_855043_52.stderr parsl.dataflow.dflow INFO: Standard output for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/2d64df3f-9441-42e0-94d1-56c0c1218f2e_isr_855043_52.stdout parsl.dataflow.dflow INFO: Standard error for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/2d64df3f-9441-42e0-94d1-56c0c1218f2e_isr_855043_52.stderr parsl.dataflow.dflow INFO: Task 879 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9ee1ec4be37e55875a425c48222133a with result from task 879 parsl.dataflow.dflow DEBUG: Task 879 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 880 has memoization hash ca94007e17e771b8d0f08a47d801081e parsl.dataflow.memoization INFO: Task 880 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45667600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 880 try 0 launched on executor multi with executor id 571 parsl.dataflow.dflow INFO: Standard output for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/39ec4d97-85f2-42b4-814d-61f21658e2fd_characterizeImage_855043_46.stdout parsl.dataflow.dflow INFO: Standard error for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/39ec4d97-85f2-42b4-814d-61f21658e2fd_characterizeImage_855043_46.stderr parsl.dataflow.dflow INFO: Standard output for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/9fc2c6c5-371b-4ccf-bb28-964d587c178f_isr_855043_46.stdout parsl.dataflow.dflow INFO: Standard error for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/9fc2c6c5-371b-4ccf-bb28-964d587c178f_isr_855043_46.stderr parsl.dataflow.dflow INFO: Task 828 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd15c4eae9fa5834ff7d5d8b59f5dd53 with result from task 828 parsl.dataflow.dflow DEBUG: Task 828 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 829 has memoization hash 1e08ee43c6c8927fb9b735390efcd87c parsl.dataflow.memoization INFO: Task 829 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45664ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 829 try 0 launched on executor multi with executor id 572 parsl.dataflow.dflow INFO: Standard output for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/b42b9a6b-38cd-4e24-94df-ff8e2a81559b_characterizeImage_855136_5.stdout parsl.dataflow.dflow INFO: Standard error for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/b42b9a6b-38cd-4e24-94df-ff8e2a81559b_characterizeImage_855136_5.stderr parsl.dataflow.dflow INFO: Standard output for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/0621f4b8-cc6d-4e59-b3be-9e1f67b45b0b_isr_855136_5.stdout parsl.dataflow.dflow INFO: Standard error for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/0621f4b8-cc6d-4e59-b3be-9e1f67b45b0b_isr_855136_5.stderr parsl.dataflow.dflow INFO: Task 387 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d91bc9a2938124bc6cd12ee26b3db86 with result from task 387 parsl.dataflow.dflow DEBUG: Task 387 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 388 has memoization hash 7683ee44c9d945153bcae6861bbea73a parsl.dataflow.memoization INFO: Task 388 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4687b7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 388 try 0 launched on executor multi with executor id 573 parsl.dataflow.dflow INFO: Standard output for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/408785e0-e2c6-45d7-bbbb-9bd43d4d2f86_characterizeImage_855042_51.stdout parsl.dataflow.dflow INFO: Standard error for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/408785e0-e2c6-45d7-bbbb-9bd43d4d2f86_characterizeImage_855042_51.stderr parsl.dataflow.dflow INFO: Standard output for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/7b34fa48-0c3f-40b2-9af3-a582b7699d2e_isr_855042_51.stdout parsl.dataflow.dflow INFO: Standard error for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/7b34fa48-0c3f-40b2-9af3-a582b7699d2e_isr_855042_51.stderr parsl.dataflow.dflow INFO: Task 867 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 99e20afc4f89beed087c8d891f31c334 with result from task 867 parsl.dataflow.dflow DEBUG: Task 867 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 868 has memoization hash 655a13d0b7b1fd8de399881421a13d19 parsl.dataflow.memoization INFO: Task 868 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45666ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 868 try 0 launched on executor multi with executor id 574 parsl.dataflow.dflow INFO: Standard output for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/c08d2d6d-b768-4316-a6e8-a08f7d04471d_characterizeImage_855044_13.stdout parsl.dataflow.dflow INFO: Standard error for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/c08d2d6d-b768-4316-a6e8-a08f7d04471d_characterizeImage_855044_13.stderr parsl.dataflow.dflow INFO: Standard output for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/4fcbab09-6512-414b-8612-275243d823b0_isr_855044_13.stdout parsl.dataflow.dflow INFO: Standard error for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/4fcbab09-6512-414b-8612-275243d823b0_isr_855044_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:22 proc_lsst.multi:146 [INFO] found job 16912310 in provider astro proc_lsst.multi INFO: found job 16912310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 909 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 677456621c830ea7532489a00a155a4d with result from task 909 parsl.dataflow.dflow DEBUG: Task 909 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 910 has memoization hash d95f1f31a56b8c0f8b8a2007d25d0948 parsl.dataflow.memoization INFO: Task 910 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45499f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 910 try 0 launched on executor multi with executor id 575 parsl.dataflow.dflow INFO: Standard output for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/e8be66b1-62e1-42c2-9ee1-fef43c057cba_characterizeImage_855043_8.stdout parsl.dataflow.dflow INFO: Standard error for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/e8be66b1-62e1-42c2-9ee1-fef43c057cba_characterizeImage_855043_8.stderr parsl.dataflow.dflow INFO: Standard output for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/64f55006-bd82-473f-9d09-c172fc733c5e_isr_855043_8.stdout parsl.dataflow.dflow INFO: Standard error for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/64f55006-bd82-473f-9d09-c172fc733c5e_isr_855043_8.stderr parsl.dataflow.dflow INFO: Task 852 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9de265bcc152cdf1a091ab026546ce5a with result from task 852 parsl.dataflow.dflow DEBUG: Task 852 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 853 has memoization hash c2129bfca05683b0c0517b023c4d390e parsl.dataflow.memoization INFO: Task 853 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45666200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 853 try 0 launched on executor multi with executor id 576 parsl.dataflow.dflow INFO: Standard output for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/edb9261f-9908-4593-b130-ca1dad030cb2_characterizeImage_855137_16.stdout parsl.dataflow.dflow INFO: Standard error for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/edb9261f-9908-4593-b130-ca1dad030cb2_characterizeImage_855137_16.stderr parsl.dataflow.dflow INFO: Standard output for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/879bea37-5690-4edb-b385-5563195f5709_isr_855137_16.stdout parsl.dataflow.dflow INFO: Standard error for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/879bea37-5690-4edb-b385-5563195f5709_isr_855137_16.stderr parsl.dataflow.dflow INFO: Task 912 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 35bfc25510fd747924802b73766f8329 with result from task 912 parsl.dataflow.dflow DEBUG: Task 912 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 913 has memoization hash 5b30b5f5a6e27b6f460e3f7529576354 parsl.dataflow.memoization INFO: Task 913 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4549a2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 913 try 0 launched on executor multi with executor id 577 parsl.dataflow.dflow INFO: Standard output for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/b7a2fe68-a718-4376-b28a-0a829652a19d_characterizeImage_855042_24.stdout parsl.dataflow.dflow INFO: Standard error for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/b7a2fe68-a718-4376-b28a-0a829652a19d_characterizeImage_855042_24.stderr parsl.dataflow.dflow INFO: Standard output for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/cd950b6b-63a1-47f7-b354-206913b10685_isr_855042_24.stdout parsl.dataflow.dflow INFO: Standard error for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/cd950b6b-63a1-47f7-b354-206913b10685_isr_855042_24.stderr parsl.dataflow.dflow INFO: Task 798 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a2f2d3d556ccfd47516f8e811ad82f8 with result from task 798 parsl.dataflow.dflow DEBUG: Task 798 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 799 has memoization hash 2cc0e6a66d9d6e52577586f3255f7d0a parsl.dataflow.memoization INFO: Task 799 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 799 try 0 launched on executor multi with executor id 578 parsl.dataflow.dflow INFO: Standard output for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/75ba3708-493d-4a90-b006-a63d1c11ceb9_characterizeImage_855138_35.stdout parsl.dataflow.dflow INFO: Standard error for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/75ba3708-493d-4a90-b006-a63d1c11ceb9_characterizeImage_855138_35.stderr parsl.dataflow.dflow INFO: Standard output for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/4ac1edb4-337e-4936-bdf7-977ab087ebe6_isr_855138_35.stdout parsl.dataflow.dflow INFO: Standard error for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/4ac1edb4-337e-4936-bdf7-977ab087ebe6_isr_855138_35.stderr parsl.dataflow.dflow INFO: Task 831 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1268c0cce88aef4390f84db7984a3d34 with result from task 831 parsl.dataflow.dflow DEBUG: Task 831 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 832 has memoization hash 5b056e220651f1047eb7667ab8af39a9 parsl.dataflow.memoization INFO: Task 832 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45664e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 832 try 0 launched on executor multi with executor id 579 parsl.dataflow.dflow INFO: Standard output for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/da31d560-b415-4448-a72c-d701f6fc457c_characterizeImage_855043_34.stdout parsl.dataflow.dflow INFO: Standard error for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/da31d560-b415-4448-a72c-d701f6fc457c_characterizeImage_855043_34.stderr parsl.dataflow.dflow INFO: Standard output for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/d8783040-41b6-4d72-a883-88d5d2efb20f_isr_855043_34.stdout parsl.dataflow.dflow INFO: Standard error for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/d8783040-41b6-4d72-a883-88d5d2efb20f_isr_855043_34.stderr parsl.dataflow.dflow INFO: Task 840 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec7b16a91b0a77b6d966330d8e4e9f38 with result from task 840 parsl.dataflow.dflow DEBUG: Task 840 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 841 has memoization hash 12527eef9962be6ba6ceeb982a921e1a parsl.dataflow.memoization INFO: Task 841 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45665580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 841 try 0 launched on executor multi with executor id 580 parsl.dataflow.dflow INFO: Standard output for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/b86aa0d6-36c8-4a4e-8071-87598f282e23_characterizeImage_855137_42.stdout parsl.dataflow.dflow INFO: Standard error for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/b86aa0d6-36c8-4a4e-8071-87598f282e23_characterizeImage_855137_42.stderr parsl.dataflow.dflow INFO: Standard output for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/6c450119-a166-4e58-a5b5-f93d7469b309_isr_855137_42.stdout parsl.dataflow.dflow INFO: Standard error for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/6c450119-a166-4e58-a5b5-f93d7469b309_isr_855137_42.stderr parsl.dataflow.dflow INFO: Task 312 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 30fc1a1393ec54bc4b9e98d5c54fbe70 with result from task 312 parsl.dataflow.dflow DEBUG: Task 312 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 313 has memoization hash 368b5a500fb3df9644db8783384bf0a6 parsl.dataflow.memoization INFO: Task 313 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 313 try 0 launched on executor multi with executor id 581 parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/879f1f6d-7610-48ad-970f-7d53090409df_characterizeImage_855042_31.stdout parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/879f1f6d-7610-48ad-970f-7d53090409df_characterizeImage_855042_31.stderr parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/f5461997-dac6-426a-b509-e5007d08cea8_isr_855042_31.stdout parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/f5461997-dac6-426a-b509-e5007d08cea8_isr_855042_31.stderr parsl.dataflow.dflow INFO: Task 357 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b2431a59f7c5e95786a3c8fbbb94bc0 with result from task 357 parsl.dataflow.dflow DEBUG: Task 357 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 358 has memoization hash 495f43acad4696df7be22976dc4efac8 parsl.dataflow.memoization INFO: Task 358 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46879a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 358 try 0 launched on executor multi with executor id 582 parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/29b429dc-034d-4303-9d84-2e4dc6ee64e7_characterizeImage_855042_34.stdout parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/29b429dc-034d-4303-9d84-2e4dc6ee64e7_characterizeImage_855042_34.stderr parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/237014a8-6b44-4d95-910a-dbf483d575d1_isr_855042_34.stdout parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/237014a8-6b44-4d95-910a-dbf483d575d1_isr_855042_34.stderr parsl.dataflow.dflow INFO: Task 318 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3caa219728dff3903411dde8b6606ab1 with result from task 318 parsl.dataflow.dflow DEBUG: Task 318 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 319 has memoization hash 361e0734c952c21a88272390a1c120cf parsl.dataflow.memoization INFO: Task 319 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 319 try 0 launched on executor multi with executor id 583 parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/ea8da8c3-49f3-48cc-a161-0fbd536d8a82_characterizeImage_855044_56.stdout parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/ea8da8c3-49f3-48cc-a161-0fbd536d8a82_characterizeImage_855044_56.stderr parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/657462b1-eecf-4d26-aee0-b6d3a5acd0f0_isr_855044_56.stdout parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/657462b1-eecf-4d26-aee0-b6d3a5acd0f0_isr_855044_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:22 proc_lsst.multi:146 [INFO] found job 16912311 in provider astro proc_lsst.multi INFO: found job 16912311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 894 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78695601c8570e5983d7a09619f06d22 with result from task 894 parsl.dataflow.dflow DEBUG: Task 894 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 895 has memoization hash 9d02c621a1ac36c4b3989846cce05d94 parsl.dataflow.memoization INFO: Task 895 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45498220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 895 try 0 launched on executor multi with executor id 584 parsl.dataflow.dflow INFO: Standard output for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/bd876c83-3e4f-43e8-8b6d-1e36b03745fa_characterizeImage_855042_25.stdout parsl.dataflow.dflow INFO: Standard error for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/bd876c83-3e4f-43e8-8b6d-1e36b03745fa_characterizeImage_855042_25.stderr parsl.dataflow.dflow INFO: Standard output for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/0a1bd9fb-13de-46da-8993-abcc822b1772_isr_855042_25.stdout parsl.dataflow.dflow INFO: Standard error for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/0a1bd9fb-13de-46da-8993-abcc822b1772_isr_855042_25.stderr parsl.dataflow.dflow INFO: Task 339 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9d430831c13ff1a7d8f7549c1e09d0c with result from task 339 parsl.dataflow.dflow DEBUG: Task 339 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 340 has memoization hash de8a9f0b2009bb63b77074655e08227d parsl.dataflow.memoization INFO: Task 340 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa468787c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 340 try 0 launched on executor multi with executor id 585 parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/fb187de7-a51f-43af-89ee-36656426f29a_characterizeImage_855137_47.stdout parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/fb187de7-a51f-43af-89ee-36656426f29a_characterizeImage_855137_47.stderr parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/51f3d2b2-e19a-4348-9333-5b5bf3449001_isr_855137_47.stdout parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/51f3d2b2-e19a-4348-9333-5b5bf3449001_isr_855137_47.stderr parsl.dataflow.dflow INFO: Task 897 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38b5538eec4b496d4fcf3c980a280190 with result from task 897 parsl.dataflow.dflow DEBUG: Task 897 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 898 has memoization hash 6c79108b3deb4faf1e4ecf97d02a7645 parsl.dataflow.memoization INFO: Task 898 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454987c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 898 try 0 launched on executor multi with executor id 586 parsl.dataflow.dflow INFO: Standard output for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/49336d7b-2f2f-40f1-a61d-f311e9602a96_characterizeImage_855136_52.stdout parsl.dataflow.dflow INFO: Standard error for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/49336d7b-2f2f-40f1-a61d-f311e9602a96_characterizeImage_855136_52.stderr parsl.dataflow.dflow INFO: Standard output for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/2bc73e82-e044-452c-89a5-709f2333e4e1_isr_855136_52.stdout parsl.dataflow.dflow INFO: Standard error for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/2bc73e82-e044-452c-89a5-709f2333e4e1_isr_855136_52.stderr parsl.dataflow.dflow INFO: Task 804 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9241ad9cb0e256bf0e3e91f8140ffff6 with result from task 804 parsl.dataflow.dflow DEBUG: Task 804 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 805 has memoization hash 37319c635b66bc57e11a43789de61a7e parsl.dataflow.memoization INFO: Task 805 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 805 try 0 launched on executor multi with executor id 587 parsl.dataflow.dflow INFO: Standard output for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/f2b0edde-d5fd-4938-84bd-5906d05715e8_characterizeImage_855136_6.stdout parsl.dataflow.dflow INFO: Standard error for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/f2b0edde-d5fd-4938-84bd-5906d05715e8_characterizeImage_855136_6.stderr parsl.dataflow.dflow INFO: Standard output for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/99250545-739d-4d7d-afa9-03cf006699a9_isr_855136_6.stdout parsl.dataflow.dflow INFO: Standard error for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/99250545-739d-4d7d-afa9-03cf006699a9_isr_855136_6.stderr parsl.dataflow.dflow INFO: Task 333 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6f3f783e69f6f02104819207ca38e88 with result from task 333 parsl.dataflow.dflow DEBUG: Task 333 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 334 has memoization hash 60285212fd19a3064d69c4710514c091 parsl.dataflow.memoization INFO: Task 334 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 334 try 0 launched on executor multi with executor id 588 parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/0460f431-375c-473f-b5e5-68d57a661e4e_characterizeImage_855138_24.stdout parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/0460f431-375c-473f-b5e5-68d57a661e4e_characterizeImage_855138_24.stderr parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/03a7c97d-77f1-473e-bd5a-4b10ad17405a_isr_855138_24.stdout parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/03a7c97d-77f1-473e-bd5a-4b10ad17405a_isr_855138_24.stderr parsl.dataflow.dflow INFO: Task 360 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ecda11f1f0d66ba0abd53315c8ecae1 with result from task 360 parsl.dataflow.dflow DEBUG: Task 360 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 361 has memoization hash 49386d940c39af8bb9b8757344a6abb7 parsl.dataflow.memoization INFO: Task 361 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46879da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 361 try 0 launched on executor multi with executor id 589 parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/e5c6ea55-f107-43ab-9850-d92da9c9c562_characterizeImage_855137_25.stdout parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/e5c6ea55-f107-43ab-9850-d92da9c9c562_characterizeImage_855137_25.stderr parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/bb4a6cb9-2152-4fcf-b12a-becdad1d33df_isr_855137_25.stdout parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/bb4a6cb9-2152-4fcf-b12a-becdad1d33df_isr_855137_25.stderr parsl.dataflow.dflow INFO: Task 330 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 40838ad3fffafff7385b0f3f672eb2de with result from task 330 parsl.dataflow.dflow DEBUG: Task 330 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 331 has memoization hash fcaea86c80afbd0603f3ae2d16834512 parsl.dataflow.memoization INFO: Task 331 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 331 try 0 launched on executor multi with executor id 590 parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/6bf80558-1bee-41cc-b8bf-bade6a0f2df4_characterizeImage_855043_62.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/6bf80558-1bee-41cc-b8bf-bade6a0f2df4_characterizeImage_855043_62.stderr parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/318800a0-6f2b-4305-a079-1b210594fa13_isr_855043_62.stdout parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/318800a0-6f2b-4305-a079-1b210594fa13_isr_855043_62.stderr parsl.dataflow.dflow INFO: Task 363 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 04a5ac1da77fedb3cd222e4f71c1f323 with result from task 363 parsl.dataflow.dflow DEBUG: Task 363 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 364 has memoization hash 288b3b06f1ac01c7ffc68a99a0e1bed5 parsl.dataflow.memoization INFO: Task 364 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4687a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 364 try 0 launched on executor multi with executor id 591 parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/e54ec542-eb96-4623-a232-9d4924953c9b_characterizeImage_855043_30.stdout parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/e54ec542-eb96-4623-a232-9d4924953c9b_characterizeImage_855043_30.stderr parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/0a1b03ca-c85f-4862-adc1-9bfb68bc8f86_isr_855043_30.stdout parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/0a1b03ca-c85f-4862-adc1-9bfb68bc8f86_isr_855043_30.stderr parsl.dataflow.dflow INFO: Task 834 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1cb4cd7301ae0db8be08b16d116891f8 with result from task 834 parsl.dataflow.dflow DEBUG: Task 834 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 835 has memoization hash 15959937fdda8819794a52c9f646a9ec parsl.dataflow.memoization INFO: Task 835 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45665120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 835 try 0 launched on executor multi with executor id 592 parsl.dataflow.dflow INFO: Standard output for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/e4414610-4f04-4805-bbfd-414f1bdd8183_characterizeImage_855044_42.stdout parsl.dataflow.dflow INFO: Standard error for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/e4414610-4f04-4805-bbfd-414f1bdd8183_characterizeImage_855044_42.stderr parsl.dataflow.dflow INFO: Standard output for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/14da00d1-4641-4a8f-b551-4246626f295e_isr_855044_42.stdout parsl.dataflow.dflow INFO: Standard error for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/14da00d1-4641-4a8f-b551-4246626f295e_isr_855044_42.stderr parsl.dataflow.dflow INFO: Task 846 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 989fcd824a4feb6d8aba5c88191a2918 with result from task 846 parsl.dataflow.dflow DEBUG: Task 846 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 847 has memoization hash a3f82bb23a084bc1728bd70d66b14710 parsl.dataflow.memoization INFO: Task 847 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45665bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 847 try 0 launched on executor multi with executor id 593 parsl.dataflow.dflow INFO: Standard output for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/c4095486-07d4-4eb1-8529-5349f41b3a3e_characterizeImage_855044_7.stdout parsl.dataflow.dflow INFO: Standard error for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/c4095486-07d4-4eb1-8529-5349f41b3a3e_characterizeImage_855044_7.stderr parsl.dataflow.dflow INFO: Standard output for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/873081f8-4899-4bc0-b4d5-e2df66f72fcb_isr_855044_7.stdout parsl.dataflow.dflow INFO: Standard error for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/873081f8-4899-4bc0-b4d5-e2df66f72fcb_isr_855044_7.stderr parsl.dataflow.dflow INFO: Task 450 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24242c8da71013e68b3c5c04f093d674 with result from task 450 parsl.dataflow.dflow DEBUG: Task 450 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 451 has memoization hash e4feacb6f2a4122e107874f4dc8969f9 parsl.dataflow.memoization INFO: Task 451 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a68e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 451 try 0 launched on executor multi with executor id 594 parsl.dataflow.dflow INFO: Standard output for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/c18d4c69-f7da-4493-ba19-0a9ba05a7692_characterizeImage_855138_9.stdout parsl.dataflow.dflow INFO: Standard error for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/c18d4c69-f7da-4493-ba19-0a9ba05a7692_characterizeImage_855138_9.stderr parsl.dataflow.dflow INFO: Standard output for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/faa5da83-1dff-4b2d-85ee-b39730ddf98c_isr_855138_9.stdout parsl.dataflow.dflow INFO: Standard error for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/faa5da83-1dff-4b2d-85ee-b39730ddf98c_isr_855138_9.stderr parsl.dataflow.dflow INFO: Task 459 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d0eb5467c34fe17af7c1d93220cd1da with result from task 459 parsl.dataflow.dflow DEBUG: Task 459 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 460 has memoization hash 1ab830f01f2c092280829c18d34b44b3 parsl.dataflow.memoization INFO: Task 460 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456cc720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:23 proc_lsst.multi:146 [INFO] found job 16912312 in provider astro proc_lsst.multi INFO: found job 16912312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Parsl task 460 try 0 launched on executor multi with executor id 595 parsl.dataflow.dflow INFO: Standard output for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/6fc0e6da-cbce-4330-8098-d3a3922aa38e_characterizeImage_855042_19.stdout parsl.dataflow.dflow INFO: Standard error for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/6fc0e6da-cbce-4330-8098-d3a3922aa38e_characterizeImage_855042_19.stderr parsl.dataflow.dflow INFO: Standard output for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/9a86b830-df18-4a82-a7bc-8e6c45b30715_isr_855042_19.stdout parsl.dataflow.dflow INFO: Standard error for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/9a86b830-df18-4a82-a7bc-8e6c45b30715_isr_855042_19.stderr parsl.dataflow.dflow INFO: Task 453 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6074fd70655549751230b0ae1f1aee60 with result from task 453 parsl.dataflow.dflow DEBUG: Task 453 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 454 has memoization hash cfa527e6a461d6dd7b848cb12fe702fa parsl.dataflow.memoization INFO: Task 454 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a7f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 454 try 0 launched on executor multi with executor id 596 parsl.dataflow.dflow INFO: Standard output for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/bb1224a5-aff1-40fd-9d5d-ac6d850b924f_characterizeImage_855137_20.stdout parsl.dataflow.dflow INFO: Standard error for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/bb1224a5-aff1-40fd-9d5d-ac6d850b924f_characterizeImage_855137_20.stderr parsl.dataflow.dflow INFO: Standard output for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/cb372d8f-c9da-4cfa-9113-c3923f0f5b43_isr_855137_20.stdout parsl.dataflow.dflow INFO: Standard error for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/cb372d8f-c9da-4cfa-9113-c3923f0f5b43_isr_855137_20.stderr parsl.dataflow.dflow INFO: Task 462 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fbbe581d079e183c7082d653ce6ec6ee with result from task 462 parsl.dataflow.dflow DEBUG: Task 462 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 463 has memoization hash c37137196978f3565a92737c00d00371 parsl.dataflow.memoization INFO: Task 463 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456cf7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 463 try 0 launched on executor multi with executor id 597 parsl.dataflow.dflow INFO: Standard output for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/217395bd-5bc4-4461-9c6e-b3b302002d85_characterizeImage_855136_40.stdout parsl.dataflow.dflow INFO: Standard error for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/217395bd-5bc4-4461-9c6e-b3b302002d85_characterizeImage_855136_40.stderr parsl.dataflow.dflow INFO: Standard output for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/e87393c0-ec2b-4720-8714-9f22009ada14_isr_855136_40.stdout parsl.dataflow.dflow INFO: Standard error for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/e87393c0-ec2b-4720-8714-9f22009ada14_isr_855136_40.stderr parsl.dataflow.dflow INFO: Task 468 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4150d9fdd9d282ddd72ca4b8ee6ae931 with result from task 468 parsl.dataflow.dflow DEBUG: Task 468 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 469 has memoization hash 2c04cd4e11cde2d117d75cf3281e76a9 parsl.dataflow.memoization INFO: Task 469 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456cf1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 469 try 0 launched on executor multi with executor id 598 parsl.dataflow.dflow INFO: Standard output for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/04eff46a-0f1f-4f66-a5c3-6403143835cd_characterizeImage_855044_15.stdout parsl.dataflow.dflow INFO: Standard error for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/04eff46a-0f1f-4f66-a5c3-6403143835cd_characterizeImage_855044_15.stderr parsl.dataflow.dflow INFO: Standard output for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/192d1b5f-f18b-464d-bb08-a5dcc976e0eb_isr_855044_15.stdout parsl.dataflow.dflow INFO: Standard error for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/192d1b5f-f18b-464d-bb08-a5dcc976e0eb_isr_855044_15.stderr parsl.dataflow.dflow INFO: Task 384 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ddc72574d3b15e885c3c80c452d5635 with result from task 384 parsl.dataflow.dflow DEBUG: Task 384 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 385 has memoization hash 26ac17feb0c9a1f2e6196a6537b6f3b9 parsl.dataflow.memoization INFO: Task 385 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4687b4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 385 try 0 launched on executor multi with executor id 599 parsl.dataflow.dflow INFO: Standard output for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/5d4a2880-f84d-4e34-9f5e-e536350eba08_characterizeImage_855136_1.stdout parsl.dataflow.dflow INFO: Standard error for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/5d4a2880-f84d-4e34-9f5e-e536350eba08_characterizeImage_855136_1.stderr parsl.dataflow.dflow INFO: Standard output for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/e0dd50af-f11d-4b1d-9956-5e98856219b9_isr_855136_1.stdout parsl.dataflow.dflow INFO: Standard error for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/e0dd50af-f11d-4b1d-9956-5e98856219b9_isr_855136_1.stderr parsl.dataflow.dflow INFO: Task 480 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80fa9a5fef4fbfc32ee64680a0511e9e with result from task 480 parsl.dataflow.dflow DEBUG: Task 480 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 481 has memoization hash b5ba960e53fc80b81f5f6629e603cc85 parsl.dataflow.memoization INFO: Task 481 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456ccf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 481 try 0 launched on executor multi with executor id 600 parsl.dataflow.dflow INFO: Standard output for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/3d4cc1dd-b196-45bc-a21b-b20a76698007_characterizeImage_855044_31.stdout parsl.dataflow.dflow INFO: Standard error for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/3d4cc1dd-b196-45bc-a21b-b20a76698007_characterizeImage_855044_31.stderr parsl.dataflow.dflow INFO: Standard output for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/989f7af6-c3a7-400a-a6f2-f6edd67cdafe_isr_855044_31.stdout parsl.dataflow.dflow INFO: Standard error for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/989f7af6-c3a7-400a-a6f2-f6edd67cdafe_isr_855044_31.stderr parsl.dataflow.dflow INFO: Task 483 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e42a90fc27e0f70c7caf8141fe646868 with result from task 483 parsl.dataflow.dflow DEBUG: Task 483 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 484 has memoization hash a7a9f5e5498c0c60505fad0c6960a1a1 parsl.dataflow.memoization INFO: Task 484 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456cd260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 484 try 0 launched on executor multi with executor id 601 parsl.dataflow.dflow INFO: Standard output for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/7269bfa7-181f-45aa-a990-1a46870a9334_characterizeImage_855136_27.stdout parsl.dataflow.dflow INFO: Standard error for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/7269bfa7-181f-45aa-a990-1a46870a9334_characterizeImage_855136_27.stderr parsl.dataflow.dflow INFO: Standard output for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/faa51076-bc80-4e3c-b2a2-9725f2d2615e_isr_855136_27.stdout parsl.dataflow.dflow INFO: Standard error for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/faa51076-bc80-4e3c-b2a2-9725f2d2615e_isr_855136_27.stderr parsl.dataflow.dflow INFO: Task 489 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc766093383995b9c425ad2bca758d67 with result from task 489 parsl.dataflow.dflow DEBUG: Task 489 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 490 has memoization hash 334ce602ef81290d307724835b670792 parsl.dataflow.memoization INFO: Task 490 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456cd760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 490 try 0 launched on executor multi with executor id 602 parsl.dataflow.dflow INFO: Standard output for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/79228e1b-7b08-4fc6-8392-89d9e7424661_characterizeImage_855044_25.stdout parsl.dataflow.dflow INFO: Standard error for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/79228e1b-7b08-4fc6-8392-89d9e7424661_characterizeImage_855044_25.stderr parsl.dataflow.dflow INFO: Standard output for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/66c0ca35-fa40-4213-a535-9e50d0ce1ad1_isr_855044_25.stdout parsl.dataflow.dflow INFO: Standard error for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/66c0ca35-fa40-4213-a535-9e50d0ce1ad1_isr_855044_25.stderr parsl.dataflow.dflow INFO: Task 474 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba1806526ce8899980204ea0e47191ed with result from task 474 parsl.dataflow.dflow DEBUG: Task 474 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 475 has memoization hash 1891a1cb03115b8a28c6b6a16f2efdd2 parsl.dataflow.memoization INFO: Task 475 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456ceb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 475 try 0 launched on executor multi with executor id 603 parsl.dataflow.dflow INFO: Standard output for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/9a381784-5938-4b22-a11c-85d1a7f61e3d_characterizeImage_855043_14.stdout parsl.dataflow.dflow INFO: Standard error for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/9a381784-5938-4b22-a11c-85d1a7f61e3d_characterizeImage_855043_14.stderr parsl.dataflow.dflow INFO: Standard output for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/cd7a6ae7-c44f-4254-a205-e852f6035894_isr_855043_14.stdout parsl.dataflow.dflow INFO: Standard error for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/cd7a6ae7-c44f-4254-a205-e852f6035894_isr_855043_14.stderr parsl.dataflow.dflow INFO: Task 300 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1067c9c6bd9ba88b4519cd6ace51bdcb with result from task 300 parsl.dataflow.dflow DEBUG: Task 300 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 301 has memoization hash 6092987c05eedc44bd4c7982de8e7db6 parsl.dataflow.memoization INFO: Task 301 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 301 try 0 launched on executor multi with executor id 604 parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/323a04f9-1e39-432a-88b0-219de3272ee3_characterizeImage_855138_53.stdout parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/323a04f9-1e39-432a-88b0-219de3272ee3_characterizeImage_855138_53.stderr parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/c4786fdf-2466-4bee-b5ac-05b8665c125a_isr_855138_53.stdout parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/c4786fdf-2466-4bee-b5ac-05b8665c125a_isr_855138_53.stderr parsl.dataflow.dflow INFO: Task 294 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 95e4bf521916d6dfce561ca98d538f7a with result from task 294 parsl.dataflow.dflow DEBUG: Task 294 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 295 has memoization hash 167de42efc8445e5627c38195d10dbd1 parsl.dataflow.memoization INFO: Task 295 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 295 try 0 launched on executor multi with executor id 605 parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/b9df9888-4af4-4cfe-b967-a8c259d0e357_characterizeImage_855138_45.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/b9df9888-4af4-4cfe-b967-a8c259d0e357_characterizeImage_855138_45.stderr parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/61f440f9-caad-47ad-aba7-6d0f224d1c6d_isr_855138_45.stdout parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/61f440f9-caad-47ad-aba7-6d0f224d1c6d_isr_855138_45.stderr parsl.dataflow.dflow INFO: Task 495 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e3eab4937669d57ae85a3c9af52544f with result from task 495 parsl.dataflow.dflow DEBUG: Task 495 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 496 has memoization hash ccd050e6609047e3469b6aab6946d982 parsl.dataflow.memoization INFO: Task 496 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456cdda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 496 try 0 launched on executor multi with executor id 606 parsl.dataflow.dflow INFO: Standard output for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/da0628c6-8ff5-4687-90d5-129e09a2251b_characterizeImage_855138_51.stdout parsl.dataflow.dflow INFO: Standard error for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/da0628c6-8ff5-4687-90d5-129e09a2251b_characterizeImage_855138_51.stderr parsl.dataflow.dflow INFO: Standard output for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/4c67464e-c01a-47f9-ab83-408b2783cbab_isr_855138_51.stdout parsl.dataflow.dflow INFO: Standard error for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/4c67464e-c01a-47f9-ab83-408b2783cbab_isr_855138_51.stderr parsl.dataflow.dflow INFO: Task 822 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f78ea92675b27af4541e2c2b7c1ba9a with result from task 822 parsl.dataflow.dflow DEBUG: Task 822 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 823 has memoization hash 54acd9c9fc4c11a5546d07ff78455f42 parsl.dataflow.memoization INFO: Task 823 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456644a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 823 try 0 launched on executor multi with executor id 607 parsl.dataflow.dflow INFO: Standard output for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/7a81f2a9-2a79-48b4-b095-477728d406a6_characterizeImage_855137_38.stdout parsl.dataflow.dflow INFO: Standard error for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/7a81f2a9-2a79-48b4-b095-477728d406a6_characterizeImage_855137_38.stderr parsl.dataflow.dflow INFO: Standard output for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/9af5649c-9507-440d-98ca-a5e0675a153b_isr_855137_38.stdout parsl.dataflow.dflow INFO: Standard error for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/9af5649c-9507-440d-98ca-a5e0675a153b_isr_855137_38.stderr parsl.dataflow.dflow INFO: Task 378 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c49d6d3238ae9fc44f294e7b9e1be66a with result from task 378 parsl.dataflow.dflow DEBUG: Task 378 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 379 has memoization hash 78710f04d2d840def8efd1c3c68bbd57 parsl.dataflow.memoization INFO: Task 379 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4687aca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 379 try 0 launched on executor multi with executor id 608 parsl.dataflow.dflow INFO: Standard output for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/9993f394-d214-4112-993b-ba80d1a62cd1_characterizeImage_855138_14.stdout parsl.dataflow.dflow INFO: Standard error for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/9993f394-d214-4112-993b-ba80d1a62cd1_characterizeImage_855138_14.stderr parsl.dataflow.dflow INFO: Standard output for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/8f6c0d83-99ee-4e8f-ab40-d3259a812e2a_isr_855138_14.stdout parsl.dataflow.dflow INFO: Standard error for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/8f6c0d83-99ee-4e8f-ab40-d3259a812e2a_isr_855138_14.stderr parsl.dataflow.dflow INFO: Task 861 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab53e4d6aec468b0150a4c19b2e084e9 with result from task 861 parsl.dataflow.dflow DEBUG: Task 861 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 862 has memoization hash f0dca257a56beed8b37078782402eaf8 parsl.dataflow.memoization INFO: Task 862 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45498e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 862 try 0 launched on executor multi with executor id 609 parsl.dataflow.dflow INFO: Standard output for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/a5a9b673-7328-4b5a-a004-daba4a61d9bd_characterizeImage_855137_53.stdout parsl.dataflow.dflow INFO: Standard error for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/a5a9b673-7328-4b5a-a004-daba4a61d9bd_characterizeImage_855137_53.stderr parsl.dataflow.dflow INFO: Standard output for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/29485926-e381-444d-963b-f3d1d782bba3_isr_855137_53.stdout parsl.dataflow.dflow INFO: Standard error for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/29485926-e381-444d-963b-f3d1d782bba3_isr_855137_53.stderr parsl.dataflow.dflow INFO: Task 348 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25057dcf7b75e65a457299685901ee79 with result from task 348 parsl.dataflow.dflow DEBUG: Task 348 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 349 has memoization hash cb0869a284e0f7ac37722ec2979bcf58 parsl.dataflow.memoization INFO: Task 349 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46879120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 349 try 0 launched on executor multi with executor id 610 parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/e2940e99-8666-4245-9f6d-fdd7edb6a5df_characterizeImage_855136_30.stdout parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/e2940e99-8666-4245-9f6d-fdd7edb6a5df_characterizeImage_855136_30.stderr parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/68c3986f-914a-46de-b21e-c11918654365_isr_855136_30.stdout parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/68c3986f-914a-46de-b21e-c11918654365_isr_855136_30.stderr parsl.dataflow.dflow INFO: Task 399 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a78cf4e2dd041e6cbec1b0f63065607 with result from task 399 parsl.dataflow.dflow DEBUG: Task 399 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 400 has memoization hash 714b50fd16878b26130714fbb823899b parsl.dataflow.memoization INFO: Task 400 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a7ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 400 try 0 launched on executor multi with executor id 611 parsl.dataflow.dflow INFO: Standard output for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/5a8c3a3a-48ee-4092-ab06-9b0527d7602c_characterizeImage_855137_27.stdout parsl.dataflow.dflow INFO: Standard error for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/5a8c3a3a-48ee-4092-ab06-9b0527d7602c_characterizeImage_855137_27.stderr parsl.dataflow.dflow INFO: Standard output for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/ff291be9-e449-4de5-93ad-ac988fbb938a_isr_855137_27.stdout parsl.dataflow.dflow INFO: Standard error for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/ff291be9-e449-4de5-93ad-ac988fbb938a_isr_855137_27.stderr parsl.dataflow.dflow INFO: Task 324 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 44912b1a2e49e457caf73ab947906abc with result from task 324 parsl.dataflow.dflow DEBUG: Task 324 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 325 has memoization hash 0b3533700f85923ab6301c0a2d518357 parsl.dataflow.memoization INFO: Task 325 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 325 try 0 launched on executor multi with executor id 612 parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/7828124f-5a80-4b84-9b93-af0c1b96b9bd_characterizeImage_855042_52.stdout parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/7828124f-5a80-4b84-9b93-af0c1b96b9bd_characterizeImage_855042_52.stderr parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/5ff64e32-dd17-400f-b8f3-8eec0a9a61b0_isr_855042_52.stdout parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/5ff64e32-dd17-400f-b8f3-8eec0a9a61b0_isr_855042_52.stderr parsl.dataflow.dflow INFO: Task 942 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d577ba2275719421ad79269f3640274d with result from task 942 parsl.dataflow.dflow DEBUG: Task 942 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 943 has memoization hash ead3f24839242e782a5aa6ce2bf8c1b2 parsl.dataflow.memoization INFO: Task 943 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454fb100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 943 try 0 launched on executor multi with executor id 613 parsl.dataflow.dflow INFO: Standard output for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/b0ee0349-a09f-4681-8d2b-8206dc46d8a0_characterizeImage_855138_22.stdout parsl.dataflow.dflow INFO: Standard error for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/b0ee0349-a09f-4681-8d2b-8206dc46d8a0_characterizeImage_855138_22.stderr parsl.dataflow.dflow INFO: Standard output for task 942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/3b460f95-20df-4725-84fe-f427d9674572_isr_855138_22.stdout parsl.dataflow.dflow INFO: Standard error for task 942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/3b460f95-20df-4725-84fe-f427d9674572_isr_855138_22.stderr parsl.dataflow.dflow INFO: Task 810 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e35c028a807fd4fedc4c75273862901b with result from task 810 parsl.dataflow.dflow DEBUG: Task 810 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 811 has memoization hash 7f212957eeafa60cd4fe2b83477efb1c parsl.dataflow.memoization INFO: Task 811 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 811 try 0 launched on executor multi with executor id 614 parsl.dataflow.dflow INFO: Standard output for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/bd65cc3a-afd4-4aca-983d-1ccd5e61622e_characterizeImage_855044_36.stdout parsl.dataflow.dflow INFO: Standard error for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/bd65cc3a-afd4-4aca-983d-1ccd5e61622e_characterizeImage_855044_36.stderr parsl.dataflow.dflow INFO: Standard output for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/806f3ede-1221-43e1-8f3e-20b0134ab696_isr_855044_36.stdout parsl.dataflow.dflow INFO: Standard error for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/806f3ede-1221-43e1-8f3e-20b0134ab696_isr_855044_36.stderr parsl.dataflow.dflow INFO: Task 900 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 373d0aca41ae3931b550ea23462ffcd8 with result from task 900 parsl.dataflow.dflow DEBUG: Task 900 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 901 has memoization hash 278fa7dd2c94bf81ac8715fffff19ffb parsl.dataflow.memoization INFO: Task 901 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454984a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 901 try 0 launched on executor multi with executor id 615 parsl.dataflow.dflow INFO: Standard output for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/c64fadd5-09f2-4722-a633-50657fd36f15_characterizeImage_855138_39.stdout parsl.dataflow.dflow INFO: Standard error for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/c64fadd5-09f2-4722-a633-50657fd36f15_characterizeImage_855138_39.stderr parsl.dataflow.dflow INFO: Standard output for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/94543380-79d2-454d-b5a6-cf95c1fa33d0_isr_855138_39.stdout parsl.dataflow.dflow INFO: Standard error for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/94543380-79d2-454d-b5a6-cf95c1fa33d0_isr_855138_39.stderr parsl.dataflow.dflow INFO: Task 354 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f41463435d3a1a5d9dc63352ef35c28 with result from task 354 parsl.dataflow.dflow DEBUG: Task 354 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 355 has memoization hash d7a74492eb9b91393247b6ff904c89e9 parsl.dataflow.memoization INFO: Task 355 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46879760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 355 try 0 launched on executor multi with executor id 616 parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/128dd7f0-3993-430b-a0b4-2241863aadad_characterizeImage_855138_10.stdout parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/128dd7f0-3993-430b-a0b4-2241863aadad_characterizeImage_855138_10.stderr parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/8a84493d-d932-4f4e-8ab9-8bc8b57a47e7_isr_855138_10.stdout parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/8a84493d-d932-4f4e-8ab9-8bc8b57a47e7_isr_855138_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:23 proc_lsst.multi:146 [INFO] found job 16912313 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912313 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.dataflow.dflow INFO: Task 465 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee08b1b6feffb8ffc0203274414d78ca with result from task 465 parsl.dataflow.dflow DEBUG: Task 465 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 466 has memoization hash ef2268178ccb9862c2bc6f168833265b parsl.dataflow.memoization INFO: Task 466 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456cf4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 466 try 0 launched on executor multi with executor id 617 parsl.dataflow.dflow INFO: Standard output for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/181b800d-7e08-4612-8667-b99f5074ad3d_characterizeImage_855042_30.stdout parsl.dataflow.dflow INFO: Standard error for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/181b800d-7e08-4612-8667-b99f5074ad3d_characterizeImage_855042_30.stderr parsl.dataflow.dflow INFO: Standard output for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/baef370b-a0b1-4d4f-9833-5265384370ac_isr_855042_30.stdout parsl.dataflow.dflow INFO: Standard error for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/baef370b-a0b1-4d4f-9833-5265384370ac_isr_855042_30.stderr parsl.dataflow.dflow INFO: Task 471 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86c0bc490e722d39ba36d83263c7956f with result from task 471 parsl.dataflow.dflow DEBUG: Task 471 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 472 has memoization hash e8764c1310a5dafd83418203cd29024b parsl.dataflow.memoization INFO: Task 472 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456cee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 472 try 0 launched on executor multi with executor id 618 parsl.dataflow.dflow INFO: Standard output for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/a745721b-67a8-4b69-842d-e03d5a48820c_characterizeImage_855042_42.stdout parsl.dataflow.dflow INFO: Standard error for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/a745721b-67a8-4b69-842d-e03d5a48820c_characterizeImage_855042_42.stderr parsl.dataflow.dflow INFO: Standard output for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/618dfb0a-8b2f-43b4-966d-b05d2d0b25a2_isr_855042_42.stdout parsl.dataflow.dflow INFO: Standard error for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/618dfb0a-8b2f-43b4-966d-b05d2d0b25a2_isr_855042_42.stderr parsl.dataflow.dflow INFO: Task 315 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4cb24fa53dfcb497868d112ce277104 with result from task 315 parsl.dataflow.dflow DEBUG: Task 315 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 316 has memoization hash df081dee189cab85646a741444afb804 parsl.dataflow.memoization INFO: Task 316 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 316 try 0 launched on executor multi with executor id 619 parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/25b31e6d-ad58-411a-a7a2-ecbb1d0aa186_characterizeImage_855043_26.stdout parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/25b31e6d-ad58-411a-a7a2-ecbb1d0aa186_characterizeImage_855043_26.stderr parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/06a77c3f-b51f-4ede-8400-ed388ade5349_isr_855043_26.stdout parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/06a77c3f-b51f-4ede-8400-ed388ade5349_isr_855043_26.stderr parsl.dataflow.dflow INFO: Task 855 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f031644239bbf45549011df31eb89e6b with result from task 855 parsl.dataflow.dflow DEBUG: Task 855 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 856 has memoization hash 75bc24e853dba4b852ad103fd3434a8f parsl.dataflow.memoization INFO: Task 856 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45499440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 856 try 0 launched on executor multi with executor id 620 parsl.dataflow.dflow INFO: Standard output for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/d8ce44fb-0353-4fd5-84a6-7e824d54df22_characterizeImage_855043_43.stdout parsl.dataflow.dflow INFO: Standard error for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/d8ce44fb-0353-4fd5-84a6-7e824d54df22_characterizeImage_855043_43.stderr parsl.dataflow.dflow INFO: Standard output for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/3de24388-ba3d-4c94-ae9d-3fb9c11f315c_isr_855043_43.stdout parsl.dataflow.dflow INFO: Standard error for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/3de24388-ba3d-4c94-ae9d-3fb9c11f315c_isr_855043_43.stderr parsl.dataflow.dflow INFO: Task 849 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69f5a8142b00f6116d1a9c4b9fe4180d with result from task 849 parsl.dataflow.dflow DEBUG: Task 849 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 850 has memoization hash e2a1dc0301cd755be9f779033a62f6f5 parsl.dataflow.memoization INFO: Task 850 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45665ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 850 try 0 launched on executor multi with executor id 621 parsl.dataflow.dflow INFO: Standard output for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/21d62c68-aa76-428e-bbe8-3ea72f81978f_characterizeImage_855042_54.stdout parsl.dataflow.dflow INFO: Standard error for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/21d62c68-aa76-428e-bbe8-3ea72f81978f_characterizeImage_855042_54.stderr parsl.dataflow.dflow INFO: Standard output for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/1e885629-62a6-49a5-aa9c-a76084bd40cb_isr_855042_54.stdout parsl.dataflow.dflow INFO: Standard error for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/1e885629-62a6-49a5-aa9c-a76084bd40cb_isr_855042_54.stderr parsl.dataflow.dflow INFO: Task 456 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 138523e560631bc9a05db2e8a6bf16a3 with result from task 456 parsl.dataflow.dflow DEBUG: Task 456 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 457 has memoization hash 073df0b1db7be7d527ffc19685f3179b parsl.dataflow.memoization INFO: Task 457 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456cc400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 457 try 0 launched on executor multi with executor id 622 parsl.dataflow.dflow INFO: Standard output for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/85300276-6776-4408-9911-8901979e9ca1_characterizeImage_855136_4.stdout parsl.dataflow.dflow INFO: Standard error for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/85300276-6776-4408-9911-8901979e9ca1_characterizeImage_855136_4.stderr parsl.dataflow.dflow INFO: Standard output for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/cfb5f3f1-7cda-4d80-b6d3-76bb8e378715_isr_855136_4.stdout parsl.dataflow.dflow INFO: Standard error for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/cfb5f3f1-7cda-4d80-b6d3-76bb8e378715_isr_855136_4.stderr parsl.dataflow.dflow INFO: Task 486 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bba9b10969f7395c013b44753e6a8316 with result from task 486 parsl.dataflow.dflow DEBUG: Task 486 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 487 has memoization hash 7db1acb168c73964a19cd57ae028411e parsl.dataflow.memoization INFO: Task 487 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456cd580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 487 try 0 launched on executor multi with executor id 623 parsl.dataflow.dflow INFO: Standard output for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/3257157e-c0c8-4704-883b-4a527a9f89e1_characterizeImage_855137_35.stdout parsl.dataflow.dflow INFO: Standard error for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/3257157e-c0c8-4704-883b-4a527a9f89e1_characterizeImage_855137_35.stderr parsl.dataflow.dflow INFO: Standard output for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/aa9e7fa8-b76e-44e9-9da8-639c3ecb2c0b_isr_855137_35.stdout parsl.dataflow.dflow INFO: Standard error for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/aa9e7fa8-b76e-44e9-9da8-639c3ecb2c0b_isr_855137_35.stderr parsl.dataflow.dflow INFO: Task 492 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 723f28b6a6c0be48c2c0671abf18f30f with result from task 492 parsl.dataflow.dflow DEBUG: Task 492 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 493 has memoization hash 55091645fbc2af2e51fec5b2b8ee6aaf parsl.dataflow.memoization INFO: Task 493 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456cda80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 493 try 0 launched on executor multi with executor id 624 parsl.dataflow.dflow INFO: Standard output for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/7023f654-6de7-447a-ba18-ec2e285d8ffc_characterizeImage_855043_3.stdout parsl.dataflow.dflow INFO: Standard error for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/7023f654-6de7-447a-ba18-ec2e285d8ffc_characterizeImage_855043_3.stderr parsl.dataflow.dflow INFO: Standard output for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/57941b70-398f-48a2-8696-41ea06152e61_isr_855043_3.stdout parsl.dataflow.dflow INFO: Standard error for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/57941b70-398f-48a2-8696-41ea06152e61_isr_855043_3.stderr parsl.dataflow.dflow INFO: Task 924 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83ab448b79951239800ddddddc4ce222 with result from task 924 parsl.dataflow.dflow DEBUG: Task 924 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 925 has memoization hash 791fd6e6b58e64604632cfff47c9089e parsl.dataflow.memoization INFO: Task 925 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45666160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 925 try 0 launched on executor multi with executor id 625 parsl.dataflow.dflow INFO: Standard output for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/eff08db8-a532-4f51-b9ce-bd03d526f6bc_characterizeImage_855138_46.stdout parsl.dataflow.dflow INFO: Standard error for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/eff08db8-a532-4f51-b9ce-bd03d526f6bc_characterizeImage_855138_46.stderr parsl.dataflow.dflow INFO: Standard output for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/5c713426-550a-4a5f-9b8e-bd9d62fdab7d_isr_855138_46.stdout parsl.dataflow.dflow INFO: Standard error for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/5c713426-550a-4a5f-9b8e-bd9d62fdab7d_isr_855138_46.stderr parsl.dataflow.dflow INFO: Task 927 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9cd237fed667841dd3b2340878acc972 with result from task 927 parsl.dataflow.dflow DEBUG: Task 927 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 928 has memoization hash 2557b1b50f460699909e9c0ae795f2ff parsl.dataflow.memoization INFO: Task 928 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4549b060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 928 try 0 launched on executor multi with executor id 626 parsl.dataflow.dflow INFO: Standard output for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/67ed7dfe-c6b6-40b4-9d69-3a7b76480482_characterizeImage_855138_25.stdout parsl.dataflow.dflow INFO: Standard error for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/67ed7dfe-c6b6-40b4-9d69-3a7b76480482_characterizeImage_855138_25.stderr parsl.dataflow.dflow INFO: Standard output for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/1116ab67-b71d-4ef4-ade9-e59c6e69beee_isr_855138_25.stdout parsl.dataflow.dflow INFO: Standard error for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/1116ab67-b71d-4ef4-ade9-e59c6e69beee_isr_855138_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:24 proc_lsst.multi:146 [INFO] found job 16912314 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912314 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.dataflow.dflow INFO: Task 807 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dfd1c10a9b1cfdda970ad929b9f47611 with result from task 807 parsl.dataflow.dflow DEBUG: Task 807 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 808 has memoization hash 8b2b7351cb68290536b10ea558a5baa3 parsl.dataflow.memoization INFO: Task 808 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 808 try 0 launched on executor multi with executor id 627 parsl.dataflow.dflow INFO: Standard output for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/da405938-444c-43fb-86d0-104588ea6697_characterizeImage_855137_58.stdout parsl.dataflow.dflow INFO: Standard error for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/da405938-444c-43fb-86d0-104588ea6697_characterizeImage_855137_58.stderr parsl.dataflow.dflow INFO: Standard output for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/615623fa-bcf0-4cad-999b-a7e91b81e038_isr_855137_58.stdout parsl.dataflow.dflow INFO: Standard error for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/615623fa-bcf0-4cad-999b-a7e91b81e038_isr_855137_58.stderr parsl.dataflow.dflow INFO: Task 396 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98b517f34d427eac1391ccaa103e7d77 with result from task 396 parsl.dataflow.dflow DEBUG: Task 396 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 397 has memoization hash 50fc7b39947cc2dd6d87e14eb24034ca parsl.dataflow.memoization INFO: Task 397 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4687a7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 397 try 0 launched on executor multi with executor id 628 parsl.dataflow.dflow INFO: Standard output for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/a78d370b-f3a7-45b8-ba3d-74d7f5c88a21_characterizeImage_855138_48.stdout parsl.dataflow.dflow INFO: Standard error for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/a78d370b-f3a7-45b8-ba3d-74d7f5c88a21_characterizeImage_855138_48.stderr parsl.dataflow.dflow INFO: Standard output for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/383e3cd7-b91a-4c25-9623-298eed50e1bb_isr_855138_48.stdout parsl.dataflow.dflow INFO: Standard error for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/383e3cd7-b91a-4c25-9623-298eed50e1bb_isr_855138_48.stderr parsl.dataflow.dflow INFO: Task 306 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b25f0ec7e57dd6b1edaeb986f485e438 with result from task 306 parsl.dataflow.dflow DEBUG: Task 306 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 307 has memoization hash 6c9fc7a70df39000b497dcce1784378b parsl.dataflow.memoization INFO: Task 307 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 307 try 0 launched on executor multi with executor id 629 parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/288f0547-cb42-4a8b-804b-071ef0cc886f_characterizeImage_855042_4.stdout parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/288f0547-cb42-4a8b-804b-071ef0cc886f_characterizeImage_855042_4.stderr parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/1b70c162-3ee5-472e-9490-de73cddc08ad_isr_855042_4.stdout parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/1b70c162-3ee5-472e-9490-de73cddc08ad_isr_855042_4.stderr parsl.dataflow.dflow INFO: Task 342 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d50d73da6f692c2c31aa4164d1850d6 with result from task 342 parsl.dataflow.dflow DEBUG: Task 342 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 343 has memoization hash c109d37563f4430fdd65f36c3d3682dc parsl.dataflow.memoization INFO: Task 343 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46878ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 343 try 0 launched on executor multi with executor id 630 parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/8aa456b0-a8df-4abc-8aa4-84d0c699ca93_characterizeImage_855136_59.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/8aa456b0-a8df-4abc-8aa4-84d0c699ca93_characterizeImage_855136_59.stderr parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/c24a396a-46ba-4584-b462-ff95a89a1e13_isr_855136_59.stdout parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/c24a396a-46ba-4584-b462-ff95a89a1e13_isr_855136_59.stderr parsl.dataflow.dflow INFO: Task 936 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 927c2a7276f1e6bdcf0965cc3b4d12dd with result from task 936 parsl.dataflow.dflow DEBUG: Task 936 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 937 has memoization hash 1035ec8e36593db0ed35beef72c06d14 parsl.dataflow.memoization INFO: Task 937 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4549b9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 937 try 0 launched on executor multi with executor id 631 parsl.dataflow.dflow INFO: Standard output for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/8757df07-08b1-4025-a193-92c2aa241ecd_characterizeImage_855138_17.stdout parsl.dataflow.dflow INFO: Standard error for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/8757df07-08b1-4025-a193-92c2aa241ecd_characterizeImage_855138_17.stderr parsl.dataflow.dflow INFO: Standard output for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/3496b1f0-35af-487c-9a9b-d5154807d2ba_isr_855138_17.stdout parsl.dataflow.dflow INFO: Standard error for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/3496b1f0-35af-487c-9a9b-d5154807d2ba_isr_855138_17.stderr parsl.dataflow.dflow INFO: Task 477 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84c2c42811faf34156d99a901be27619 with result from task 477 parsl.dataflow.dflow DEBUG: Task 477 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 478 has memoization hash 7dd3bbc8d7a07566c0f7615d9bcc70fc parsl.dataflow.memoization INFO: Task 478 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456cc9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 478 try 0 launched on executor multi with executor id 632 parsl.dataflow.dflow INFO: Standard output for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/be51a605-79de-4a6a-8ea5-f93ec2f4e381_characterizeImage_855043_60.stdout parsl.dataflow.dflow INFO: Standard error for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/be51a605-79de-4a6a-8ea5-f93ec2f4e381_characterizeImage_855043_60.stderr parsl.dataflow.dflow INFO: Standard output for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/97dd0409-b671-4190-b110-63bc6e6caec2_isr_855043_60.stdout parsl.dataflow.dflow INFO: Standard error for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/97dd0409-b671-4190-b110-63bc6e6caec2_isr_855043_60.stderr parsl.dataflow.dflow INFO: Task 351 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a725b47e7c2b27eca9cb965423ee502 with result from task 351 parsl.dataflow.dflow DEBUG: Task 351 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 352 has memoization hash f3ecd7a9fb332051b486abcd58e8a5c5 parsl.dataflow.memoization INFO: Task 352 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46879440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 352 try 0 launched on executor multi with executor id 633 parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/a7fdd3dd-173a-4bc0-b2b6-d9c8436cabe9_characterizeImage_855042_38.stdout parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/a7fdd3dd-173a-4bc0-b2b6-d9c8436cabe9_characterizeImage_855042_38.stderr parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/d4dcfe6f-ad44-4575-95ab-41ffd1a42e95_isr_855042_38.stdout parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/d4dcfe6f-ad44-4575-95ab-41ffd1a42e95_isr_855042_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:24 proc_lsst.multi:146 [INFO] found job 16912315 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912315 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.dataflow.dflow INFO: Task 864 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03fcaede4957643efe11ef6566b27d60 with result from task 864 parsl.dataflow.dflow DEBUG: Task 864 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 865 has memoization hash b3fa3b170e8b6e1ab4574fd6eb14e54b parsl.dataflow.memoization INFO: Task 865 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456665c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 865 try 0 launched on executor multi with executor id 634 parsl.dataflow.dflow INFO: Standard output for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/11641d94-4cd5-4f70-96d4-b32e73f70a06_characterizeImage_855136_61.stdout parsl.dataflow.dflow INFO: Standard error for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/11641d94-4cd5-4f70-96d4-b32e73f70a06_characterizeImage_855136_61.stderr parsl.dataflow.dflow INFO: Standard output for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/57bd1a28-f107-4409-bf78-61588090f5bd_isr_855136_61.stdout parsl.dataflow.dflow INFO: Standard error for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/57bd1a28-f107-4409-bf78-61588090f5bd_isr_855136_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:25 proc_lsst.multi:146 [INFO] found job 16912316 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912316 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.dataflow.dflow INFO: Task 717 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ee5f350526d455ea294562c977627d3 with result from task 717 parsl.dataflow.dflow DEBUG: Task 717 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 718 has memoization hash 6f8d768665ddaeca389b8156a17b941e parsl.dataflow.memoization INFO: Task 718 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d14e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 718 try 0 launched on executor multi with executor id 635 parsl.dataflow.dflow INFO: Standard output for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/c9f38fef-ae51-4422-8a5b-af51f4a87389_characterizeImage_855042_6.stdout parsl.dataflow.dflow INFO: Standard error for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/c9f38fef-ae51-4422-8a5b-af51f4a87389_characterizeImage_855042_6.stderr parsl.dataflow.dflow INFO: Standard output for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/1b7ae0d4-a839-4e4c-bb9a-bbb1dc545a89_isr_855042_6.stdout parsl.dataflow.dflow INFO: Standard error for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/1b7ae0d4-a839-4e4c-bb9a-bbb1dc545a89_isr_855042_6.stderr parsl.dataflow.dflow INFO: Task 732 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a650a0d6e4b0577646afa360cc4fb78 with result from task 732 parsl.dataflow.dflow DEBUG: Task 732 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 733 has memoization hash e2b18739d0bc8a780e844a0231176129 parsl.dataflow.memoization INFO: Task 733 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d2480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 733 try 0 launched on executor multi with executor id 636 parsl.dataflow.dflow INFO: Standard output for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/912de327-6e6b-45e9-bb53-7af41f1e4b5b_characterizeImage_855136_29.stdout parsl.dataflow.dflow INFO: Standard error for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/912de327-6e6b-45e9-bb53-7af41f1e4b5b_characterizeImage_855136_29.stderr parsl.dataflow.dflow INFO: Standard output for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/7e4f9d0d-4594-4076-bc1f-0df79901e131_isr_855136_29.stdout parsl.dataflow.dflow INFO: Standard error for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/7e4f9d0d-4594-4076-bc1f-0df79901e131_isr_855136_29.stderr parsl.dataflow.dflow INFO: Task 714 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a519d7c96d508fdad4ee7d85f0a6df47 with result from task 714 parsl.dataflow.dflow DEBUG: Task 714 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 715 has memoization hash 609d67b52a9a201a4d262247f40d532c parsl.dataflow.memoization INFO: Task 715 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 715 try 0 launched on executor multi with executor id 637 parsl.dataflow.dflow INFO: Standard output for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/31a9a9a9-ebfd-4fb0-a3ab-c43e98b3cb1e_characterizeImage_855137_12.stdout parsl.dataflow.dflow INFO: Standard error for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/31a9a9a9-ebfd-4fb0-a3ab-c43e98b3cb1e_characterizeImage_855137_12.stderr parsl.dataflow.dflow INFO: Standard output for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/8b827292-dac9-4f4b-b09d-ab57f1190a70_isr_855137_12.stdout parsl.dataflow.dflow INFO: Standard error for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/8b827292-dac9-4f4b-b09d-ab57f1190a70_isr_855137_12.stderr parsl.dataflow.dflow INFO: Task 729 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ea3a159c3b953bfa3a78e8a5c3aff8f with result from task 729 parsl.dataflow.dflow DEBUG: Task 729 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 730 has memoization hash cbc233fedaea1ed123de11cbf3048a49 parsl.dataflow.memoization INFO: Task 730 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d2160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 730 try 0 launched on executor multi with executor id 638 parsl.dataflow.dflow INFO: Standard output for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/12f32eb7-6d29-4620-88e7-b28c0ea271fd_characterizeImage_855044_47.stdout parsl.dataflow.dflow INFO: Standard error for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/12f32eb7-6d29-4620-88e7-b28c0ea271fd_characterizeImage_855044_47.stderr parsl.dataflow.dflow INFO: Standard output for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/9b9bb813-bc72-48c6-9249-a2938d641ea7_isr_855044_47.stdout parsl.dataflow.dflow INFO: Standard error for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/9b9bb813-bc72-48c6-9249-a2938d641ea7_isr_855044_47.stderr parsl.dataflow.dflow INFO: Task 726 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89d41e6c553e1b133f5939c3d967f0d2 with result from task 726 parsl.dataflow.dflow DEBUG: Task 726 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 727 has memoization hash aeaf0a94fc91c8f27d216425ddb33096 parsl.dataflow.memoization INFO: Task 727 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d1e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 727 try 0 launched on executor multi with executor id 639 parsl.dataflow.dflow INFO: Standard output for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/05c92e93-6692-410c-808c-7e274b772fb3_characterizeImage_855042_41.stdout parsl.dataflow.dflow INFO: Standard error for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/05c92e93-6692-410c-808c-7e274b772fb3_characterizeImage_855042_41.stderr parsl.dataflow.dflow INFO: Standard output for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/fad4bc0f-b2ee-47c0-ad9d-285c4dd09eb5_isr_855042_41.stdout parsl.dataflow.dflow INFO: Standard error for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/fad4bc0f-b2ee-47c0-ad9d-285c4dd09eb5_isr_855042_41.stderr parsl.dataflow.dflow INFO: Task 720 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3f265cd67c5e4a0f42b62574473bab0 with result from task 720 parsl.dataflow.dflow DEBUG: Task 720 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 721 has memoization hash c1ebd70dcfed6b4f9ceee399d2f1079d parsl.dataflow.memoization INFO: Task 721 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d1800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 721 try 0 launched on executor multi with executor id 640 parsl.dataflow.dflow INFO: Standard output for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/31732b93-acf4-4eac-bc11-a37675124968_characterizeImage_855044_9.stdout parsl.dataflow.dflow INFO: Standard error for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/31732b93-acf4-4eac-bc11-a37675124968_characterizeImage_855044_9.stderr parsl.dataflow.dflow INFO: Standard output for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/241df1ee-03d1-4e39-8ec2-639555fd0321_isr_855044_9.stdout parsl.dataflow.dflow INFO: Standard error for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/241df1ee-03d1-4e39-8ec2-639555fd0321_isr_855044_9.stderr parsl.dataflow.dflow INFO: Task 711 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94f7c6fbccdf9229dc431041e63835d2 with result from task 711 parsl.dataflow.dflow DEBUG: Task 711 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 712 has memoization hash f75048fe2dcc84b871bfacf5324a5817 parsl.dataflow.memoization INFO: Task 712 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d1080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 712 try 0 launched on executor multi with executor id 641 parsl.dataflow.dflow INFO: Standard output for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/5f30893a-935f-41dc-9447-6fb33ccac56e_characterizeImage_855138_27.stdout parsl.dataflow.dflow INFO: Standard error for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/5f30893a-935f-41dc-9447-6fb33ccac56e_characterizeImage_855138_27.stderr parsl.dataflow.dflow INFO: Standard output for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/8c4cc0e7-28ae-49d2-a27e-a29cc052d637_isr_855138_27.stdout parsl.dataflow.dflow INFO: Standard error for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/8c4cc0e7-28ae-49d2-a27e-a29cc052d637_isr_855138_27.stderr parsl.dataflow.dflow INFO: Task 951 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab2b4893a643b0bb51f278a0e3fa6e89 with result from task 951 parsl.dataflow.dflow DEBUG: Task 951 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 952 has memoization hash 34d178d8ff3492b19f8e9af80b1bd71f parsl.dataflow.memoization INFO: Task 952 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454fa480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 952 try 0 launched on executor multi with executor id 642 parsl.dataflow.dflow INFO: Standard output for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/f042c185-8882-44a5-bf17-2f304c8bbe54_characterizeImage_855042_11.stdout parsl.dataflow.dflow INFO: Standard error for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/f042c185-8882-44a5-bf17-2f304c8bbe54_characterizeImage_855042_11.stderr parsl.dataflow.dflow INFO: Standard output for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/6eae4fe6-04c6-49c7-95cd-4827cd27f062_isr_855042_11.stdout parsl.dataflow.dflow INFO: Standard error for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/6eae4fe6-04c6-49c7-95cd-4827cd27f062_isr_855042_11.stderr parsl.dataflow.dflow INFO: Task 939 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a906d609c4de5880569be70617bf39a0 with result from task 939 parsl.dataflow.dflow DEBUG: Task 939 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 940 has memoization hash 0d8a4739a7907782958ac5c3bc42a497 parsl.dataflow.memoization INFO: Task 940 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4549bce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 940 try 0 launched on executor multi with executor id 643 parsl.dataflow.dflow INFO: Standard output for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/2a0661d8-8ed5-4449-906e-10c5822d43da_characterizeImage_855138_30.stdout parsl.dataflow.dflow INFO: Standard error for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/2a0661d8-8ed5-4449-906e-10c5822d43da_characterizeImage_855138_30.stderr parsl.dataflow.dflow INFO: Standard output for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/9d502c2e-a165-4311-abd0-1e86df1685f8_isr_855138_30.stdout parsl.dataflow.dflow INFO: Standard error for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/9d502c2e-a165-4311-abd0-1e86df1685f8_isr_855138_30.stderr parsl.dataflow.dflow INFO: Task 930 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4972e3e94839bdcc6cfddfc783612622 with result from task 930 parsl.dataflow.dflow DEBUG: Task 930 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 931 has memoization hash b4825bdc9fa9e2fc1e5cbc57c93c8def parsl.dataflow.memoization INFO: Task 931 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4549b380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 931 try 0 launched on executor multi with executor id 644 parsl.dataflow.dflow INFO: Standard output for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/b2eed188-6b29-4e8d-9d75-c0b6dd463507_characterizeImage_855136_55.stdout parsl.dataflow.dflow INFO: Standard error for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/b2eed188-6b29-4e8d-9d75-c0b6dd463507_characterizeImage_855136_55.stderr parsl.dataflow.dflow INFO: Standard output for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/47f067b7-1c03-4e71-ae3e-caefbf9f0305_isr_855136_55.stdout parsl.dataflow.dflow INFO: Standard error for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/47f067b7-1c03-4e71-ae3e-caefbf9f0305_isr_855136_55.stderr parsl.dataflow.dflow INFO: Task 723 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ef80493ebb41925942dfb9e16254e6c with result from task 723 parsl.dataflow.dflow DEBUG: Task 723 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 724 has memoization hash dbce7b778b3d9096ce9635dbc9e157d4 parsl.dataflow.memoization INFO: Task 724 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d1b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 724 try 0 launched on executor multi with executor id 645 parsl.dataflow.dflow INFO: Standard output for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/c5067539-eb35-47f6-bd36-f9909c37b581_characterizeImage_855138_12.stdout parsl.dataflow.dflow INFO: Standard error for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/c5067539-eb35-47f6-bd36-f9909c37b581_characterizeImage_855138_12.stderr parsl.dataflow.dflow INFO: Standard output for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/e232475a-8f16-4236-9f66-c84fc5828b0b_isr_855138_12.stdout parsl.dataflow.dflow INFO: Standard error for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/e232475a-8f16-4236-9f66-c84fc5828b0b_isr_855138_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:25 proc_lsst.multi:146 [INFO] found job 16912317 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.dataflow.dflow INFO: Task 957 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98c7eb63b1c5356a33da96e1012ed4d8 with result from task 957 parsl.dataflow.dflow DEBUG: Task 957 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 958 has memoization hash d29d89c96f3844e9df48fc39e41da473 parsl.dataflow.memoization INFO: Task 958 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454f8040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 958 try 0 launched on executor multi with executor id 646 parsl.dataflow.dflow INFO: Standard output for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/464e5e09-2ad3-41b6-896c-c8e625e2f148_characterizeImage_855137_22.stdout parsl.dataflow.dflow INFO: Standard error for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/464e5e09-2ad3-41b6-896c-c8e625e2f148_characterizeImage_855137_22.stderr parsl.dataflow.dflow INFO: Standard output for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/f9835ac6-0f43-47c8-b83e-f9fabb593841_isr_855137_22.stdout parsl.dataflow.dflow INFO: Standard error for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/f9835ac6-0f43-47c8-b83e-f9fabb593841_isr_855137_22.stderr parsl.dataflow.dflow INFO: Task 960 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a649b524262c9138b618eb64f598ee5b with result from task 960 parsl.dataflow.dflow DEBUG: Task 960 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 961 has memoization hash cc3bed2fd567b0a4982ade5dbc6d3f09 parsl.dataflow.memoization INFO: Task 961 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454f8360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 961 try 0 launched on executor multi with executor id 647 parsl.dataflow.dflow INFO: Standard output for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/e4d6d4c2-63dd-4dda-95d6-476aa224b354_characterizeImage_855043_27.stdout parsl.dataflow.dflow INFO: Standard error for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/e4d6d4c2-63dd-4dda-95d6-476aa224b354_characterizeImage_855043_27.stderr parsl.dataflow.dflow INFO: Standard output for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/542c9f54-dcc9-4d14-a31c-c5a43324fe0e_isr_855043_27.stdout parsl.dataflow.dflow INFO: Standard error for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/542c9f54-dcc9-4d14-a31c-c5a43324fe0e_isr_855043_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:25 proc_lsst.multi:146 [INFO] found job 16912318 in provider escience_a40 proc_lsst.multi INFO: found job 16912318 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 327 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9658da980f64c9d3eca07a333344a606 with result from task 327 parsl.dataflow.dflow DEBUG: Task 327 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 328 has memoization hash e25904ffa816835242d3f9d89f3bb963 parsl.dataflow.memoization INFO: Task 328 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 328 try 0 launched on executor multi with executor id 648 parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/44bed20b-ca1d-4cbd-aa34-7d2c2a5f3e22_characterizeImage_855138_58.stdout parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/44bed20b-ca1d-4cbd-aa34-7d2c2a5f3e22_characterizeImage_855138_58.stderr parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/68543290-cc20-45dc-a640-55869cb9834b_isr_855138_58.stdout parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/68543290-cc20-45dc-a640-55869cb9834b_isr_855138_58.stderr parsl.dataflow.dflow INFO: Task 948 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b618131f3228a0071191b97bc78d77f with result from task 948 parsl.dataflow.dflow DEBUG: Task 948 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 949 has memoization hash 5cbf1bee8d85e492d5cdf2162e2408bf parsl.dataflow.memoization INFO: Task 949 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454fa7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 949 try 0 launched on executor multi with executor id 649 parsl.dataflow.dflow INFO: Standard output for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/07263472-46b8-4b80-8985-a0d1bde7c412_characterizeImage_855136_58.stdout parsl.dataflow.dflow INFO: Standard error for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/07263472-46b8-4b80-8985-a0d1bde7c412_characterizeImage_855136_58.stderr parsl.dataflow.dflow INFO: Standard output for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/584fa0d8-1059-4183-99a3-ebf4765a5285_isr_855136_58.stdout parsl.dataflow.dflow INFO: Standard error for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/584fa0d8-1059-4183-99a3-ebf4765a5285_isr_855136_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:26 proc_lsst.multi:146 [INFO] found job 16912319 in provider escience_a40 proc_lsst.multi INFO: found job 16912319 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 945 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4013b9a464b599cca22b3bf190be95ba with result from task 945 parsl.dataflow.dflow DEBUG: Task 945 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 946 has memoization hash d739f5c00e671c8ba55842105e1fffd1 parsl.dataflow.memoization INFO: Task 946 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454faac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 946 try 0 launched on executor multi with executor id 650 parsl.dataflow.dflow INFO: Standard output for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/f18a6e67-38ad-4947-8533-fb1868ed225c_characterizeImage_855136_22.stdout parsl.dataflow.dflow INFO: Standard error for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/f18a6e67-38ad-4947-8533-fb1868ed225c_characterizeImage_855136_22.stderr parsl.dataflow.dflow INFO: Standard output for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/eed15330-fa29-486a-8334-e80ec6c94326_isr_855136_22.stdout parsl.dataflow.dflow INFO: Standard error for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/eed15330-fa29-486a-8334-e80ec6c94326_isr_855136_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:26 proc_lsst.multi:146 [INFO] found job 16912320 in provider escience_a40 proc_lsst.multi INFO: found job 16912320 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 345 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8819ecdd3547b4ebb31079f6efdefb26 with result from task 345 parsl.dataflow.dflow DEBUG: Task 345 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 346 has memoization hash de5d53d135a5a71bec336ffb260c2831 parsl.dataflow.memoization INFO: Task 346 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46878e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 346 try 0 launched on executor multi with executor id 651 parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/aa5d6fda-321c-4e44-b9f2-76d10a9df141_characterizeImage_855043_25.stdout parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/aa5d6fda-321c-4e44-b9f2-76d10a9df141_characterizeImage_855043_25.stderr parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/ea5cde7a-91f6-4106-9a40-8dac767c91ff_isr_855043_25.stdout parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/ea5cde7a-91f6-4106-9a40-8dac767c91ff_isr_855043_25.stderr parsl.dataflow.dflow INFO: Task 309 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e1365ab112194bb27e48dec2bde3c7b with result from task 309 parsl.dataflow.dflow DEBUG: Task 309 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 310 has memoization hash 69bae4c0ae94bab67f0f1396df58298e parsl.dataflow.memoization INFO: Task 310 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 310 try 0 launched on executor multi with executor id 652 parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/0f76515e-3707-43d8-afa2-41f8f832e03e_characterizeImage_855043_42.stdout parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/0f76515e-3707-43d8-afa2-41f8f832e03e_characterizeImage_855043_42.stderr parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/ea61665c-dd14-4a61-98c6-a60446d78eeb_isr_855043_42.stdout parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/ea61665c-dd14-4a61-98c6-a60446d78eeb_isr_855043_42.stderr parsl.dataflow.dflow INFO: Task 954 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4661f4f20defc6fea74643189fb9b58d with result from task 954 parsl.dataflow.dflow DEBUG: Task 954 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 955 has memoization hash 5cee067dece8740030789b3c9469aad8 parsl.dataflow.memoization INFO: Task 955 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4549be20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 955 try 0 launched on executor multi with executor id 653 parsl.dataflow.dflow INFO: Standard output for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/d6440207-a7f8-4b3d-a85f-e61cb04f150d_characterizeImage_855138_7.stdout parsl.dataflow.dflow INFO: Standard error for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/d6440207-a7f8-4b3d-a85f-e61cb04f150d_characterizeImage_855138_7.stderr parsl.dataflow.dflow INFO: Standard output for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/a1e9cd58-8335-4862-ad93-47a0d0e69386_isr_855138_7.stdout parsl.dataflow.dflow INFO: Standard error for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/a1e9cd58-8335-4862-ad93-47a0d0e69386_isr_855138_7.stderr parsl.dataflow.dflow INFO: Task 987 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ddcbdef52c365e6b1635502d4aa1bbf with result from task 987 parsl.dataflow.dflow DEBUG: Task 987 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 988 has memoization hash 24bba4fdeb7378f98cb7c2daf6aacfdb parsl.dataflow.memoization INFO: Task 988 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454f9e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 988 try 0 launched on executor multi with executor id 654 parsl.dataflow.dflow INFO: Standard output for task 988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/09fbcbaa-9568-4ab1-8dfa-7ceb1d2b3af1_characterizeImage_855136_51.stdout parsl.dataflow.dflow INFO: Standard error for task 988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/09fbcbaa-9568-4ab1-8dfa-7ceb1d2b3af1_characterizeImage_855136_51.stderr parsl.dataflow.dflow INFO: Standard output for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/e569cab5-ef45-4494-b5c6-d272b9adca35_isr_855136_51.stdout parsl.dataflow.dflow INFO: Standard error for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/e569cab5-ef45-4494-b5c6-d272b9adca35_isr_855136_51.stderr parsl.dataflow.dflow INFO: Task 981 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5db215b4e9a95aa6fec0d01324a1b81a with result from task 981 parsl.dataflow.dflow DEBUG: Task 981 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 982 has memoization hash 4ddb82f0f133712abf8410789907a113 parsl.dataflow.memoization INFO: Task 982 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454f99e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 982 try 0 launched on executor multi with executor id 655 parsl.dataflow.dflow INFO: Standard output for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/9d644d4e-ecb9-4d8c-b1dc-3aabaf34b0d7_characterizeImage_855138_31.stdout parsl.dataflow.dflow INFO: Standard error for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/9d644d4e-ecb9-4d8c-b1dc-3aabaf34b0d7_characterizeImage_855138_31.stderr parsl.dataflow.dflow INFO: Standard output for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/f711336e-1e86-456f-8049-0bb08264fb48_isr_855138_31.stdout parsl.dataflow.dflow INFO: Standard error for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/f711336e-1e86-456f-8049-0bb08264fb48_isr_855138_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:27 proc_lsst.multi:146 [INFO] found job 16912321 in provider escience_a40 proc_lsst.multi INFO: found job 16912321 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 933 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 645a2ea5a3385df82d178f943c2852f5 with result from task 933 parsl.dataflow.dflow DEBUG: Task 933 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 934 has memoization hash c32c43637ca0b2584f010862d16b525f parsl.dataflow.memoization INFO: Task 934 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4549b6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 934 try 0 launched on executor multi with executor id 656 parsl.dataflow.dflow INFO: Standard output for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/dcd9040f-7e9a-4980-ad67-e65ecbd76a42_characterizeImage_855137_54.stdout parsl.dataflow.dflow INFO: Standard error for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/dcd9040f-7e9a-4980-ad67-e65ecbd76a42_characterizeImage_855137_54.stderr parsl.dataflow.dflow INFO: Standard output for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/829d50d6-3168-4350-a68d-e2babdf13c00_isr_855137_54.stdout parsl.dataflow.dflow INFO: Standard error for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/829d50d6-3168-4350-a68d-e2babdf13c00_isr_855137_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:27 proc_lsst.multi:146 [INFO] found job 16912322 in provider escience_a40 proc_lsst.multi INFO: found job 16912322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 303 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05a30b1d584a71b262571f01884274ac with result from task 303 parsl.dataflow.dflow DEBUG: Task 303 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 304 has memoization hash c1d72151abb9626448c8320a1353a6d0 parsl.dataflow.memoization INFO: Task 304 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 304 try 0 launched on executor multi with executor id 657 parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/178c32e4-7522-418d-8d08-c6be0d08a4c9_characterizeImage_855044_39.stdout parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/178c32e4-7522-418d-8d08-c6be0d08a4c9_characterizeImage_855044_39.stderr parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/0d37e399-d0b5-4285-b29a-fee8bb9f55dc_isr_855044_39.stdout parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/0d37e399-d0b5-4285-b29a-fee8bb9f55dc_isr_855044_39.stderr parsl.dataflow.dflow INFO: Task 921 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 40946beb15cbd1574d2bc0d3f90d8226 with result from task 921 parsl.dataflow.dflow DEBUG: Task 921 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 922 has memoization hash 8ab9dc8f962eafb6e49e897bb9a8d50d parsl.dataflow.memoization INFO: Task 922 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4549ac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 922 try 0 launched on executor multi with executor id 658 parsl.dataflow.dflow INFO: Standard output for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/344e994b-7936-421f-b2b3-7b242ac710ba_characterizeImage_855044_55.stdout parsl.dataflow.dflow INFO: Standard error for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/344e994b-7936-421f-b2b3-7b242ac710ba_characterizeImage_855044_55.stderr parsl.dataflow.dflow INFO: Standard output for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/6d3fd601-7bd5-4aab-8683-62ba84837909_isr_855044_55.stdout parsl.dataflow.dflow INFO: Standard error for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/6d3fd601-7bd5-4aab-8683-62ba84837909_isr_855044_55.stderr parsl.dataflow.dflow INFO: Task 297 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d06a5831e386b28ac99d30c6deb61fa2 with result from task 297 parsl.dataflow.dflow DEBUG: Task 297 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 298 has memoization hash 481479a7d79c0c16eb43ca2e298ee348 parsl.dataflow.memoization INFO: Task 298 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 298 try 0 launched on executor multi with executor id 659 parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/2196edd3-8030-41f7-a046-36643ff87b2a_characterizeImage_855137_43.stdout parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/2196edd3-8030-41f7-a046-36643ff87b2a_characterizeImage_855137_43.stderr parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/eaa29b7d-dc64-41c5-810f-f6b2e4c11b8d_isr_855137_43.stdout parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/eaa29b7d-dc64-41c5-810f-f6b2e4c11b8d_isr_855137_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:27 proc_lsst.multi:146 [INFO] found job 16912323 in provider escience_a40 proc_lsst.multi INFO: found job 16912323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 381 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e979cf75ee5090df46477ec925413d1e with result from task 381 parsl.dataflow.dflow DEBUG: Task 381 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 382 has memoization hash 2e77257b064c2824f4a646aa40bf7f4b parsl.dataflow.memoization INFO: Task 382 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4687b100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 382 try 0 launched on executor multi with executor id 660 parsl.dataflow.dflow INFO: Standard output for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/548a6c7f-2d56-4e95-98da-ff264dbc9124_characterizeImage_855042_59.stdout parsl.dataflow.dflow INFO: Standard error for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/548a6c7f-2d56-4e95-98da-ff264dbc9124_characterizeImage_855042_59.stderr parsl.dataflow.dflow INFO: Standard output for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/46ccc8e2-07bd-4a50-a91c-d68b6114983b_isr_855042_59.stdout parsl.dataflow.dflow INFO: Standard error for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/46ccc8e2-07bd-4a50-a91c-d68b6114983b_isr_855042_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:28 proc_lsst.multi:146 [INFO] found job 16912324 in provider escience_a40 proc_lsst.multi INFO: found job 16912324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 966 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1185578ebcfe468af0816bda25e8e17 with result from task 966 parsl.dataflow.dflow DEBUG: Task 966 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 967 has memoization hash 7cac6dabdefa7c212efc50a0d445d397 parsl.dataflow.memoization INFO: Task 967 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454f87c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 967 try 0 launched on executor multi with executor id 661 parsl.dataflow.dflow INFO: Standard output for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/2d68c8c5-937c-45a1-9ea5-03f002e0a657_characterizeImage_855043_7.stdout parsl.dataflow.dflow INFO: Standard error for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/2d68c8c5-937c-45a1-9ea5-03f002e0a657_characterizeImage_855043_7.stderr parsl.dataflow.dflow INFO: Standard output for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/1547185b-6c45-4697-80a3-14921a62b21c_isr_855043_7.stdout parsl.dataflow.dflow INFO: Standard error for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/1547185b-6c45-4697-80a3-14921a62b21c_isr_855043_7.stderr parsl.dataflow.dflow INFO: Task 963 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2cbf920d1b4eb6d769bfbbc92f0230ee with result from task 963 parsl.dataflow.dflow DEBUG: Task 963 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 964 has memoization hash 1085ca6a8afd1995f01483c137c4e0f9 parsl.dataflow.memoization INFO: Task 964 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454f8680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 964 try 0 launched on executor multi with executor id 662 parsl.dataflow.dflow INFO: Standard output for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/66e0d236-d259-4777-86af-60f4dadf04fe_characterizeImage_855043_54.stdout parsl.dataflow.dflow INFO: Standard error for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/66e0d236-d259-4777-86af-60f4dadf04fe_characterizeImage_855043_54.stderr parsl.dataflow.dflow INFO: Standard output for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/55c3a70a-4f39-446c-b1da-66ad4abf9b63_isr_855043_54.stdout parsl.dataflow.dflow INFO: Standard error for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/55c3a70a-4f39-446c-b1da-66ad4abf9b63_isr_855043_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:28 proc_lsst.multi:146 [INFO] found job 16912325 in provider escience_a40 proc_lsst.multi INFO: found job 16912325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 990 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b065f6c4e7eddb8155678a36ee08f15e with result from task 990 parsl.dataflow.dflow DEBUG: Task 990 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 991 has memoization hash 78f4118382e357fe9ddb28bd3de3ec88 parsl.dataflow.memoization INFO: Task 991 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454fb420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 991 try 0 launched on executor multi with executor id 663 parsl.dataflow.dflow INFO: Standard output for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/a29d6546-3790-4c37-abb1-17e5e204d174_characterizeImage_855044_11.stdout parsl.dataflow.dflow INFO: Standard error for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/a29d6546-3790-4c37-abb1-17e5e204d174_characterizeImage_855044_11.stderr parsl.dataflow.dflow INFO: Standard output for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/2972c186-87ef-4341-90b8-7f3a60c72d01_isr_855044_11.stdout parsl.dataflow.dflow INFO: Standard error for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/2972c186-87ef-4341-90b8-7f3a60c72d01_isr_855044_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:28 proc_lsst.multi:146 [INFO] found job 16912326 in provider escience_a40 proc_lsst.multi INFO: found job 16912326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 1068 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a1656c4c4ba39497a6ff916f34b95c7 with result from task 1068 parsl.dataflow.dflow DEBUG: Task 1068 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1069 has memoization hash 3ddef1343a806f2ce22a60da125fc193 parsl.dataflow.memoization INFO: Task 1069 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa453931a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1069 try 0 launched on executor multi with executor id 664 parsl.dataflow.dflow INFO: Standard output for task 1069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/8fbb9451-4989-4975-8402-360d1c4d67df_characterizeImage_855138_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/8fbb9451-4989-4975-8402-360d1c4d67df_characterizeImage_855138_36.stderr parsl.dataflow.dflow INFO: Standard output for task 1068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/de1ff9c8-a419-4670-a93b-5817763c7be9_isr_855138_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/de1ff9c8-a419-4670-a93b-5817763c7be9_isr_855138_36.stderr parsl.dataflow.dflow INFO: Task 993 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0155eb076a73c0b86dc3ea68c26e57bf with result from task 993 parsl.dataflow.dflow DEBUG: Task 993 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 994 has memoization hash 335d10d1d33a3afde4dab0c9ffe2a3d5 parsl.dataflow.memoization INFO: Task 994 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454fb740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 994 try 0 launched on executor multi with executor id 665 parsl.dataflow.dflow INFO: Standard output for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/85d0f654-ffcc-478a-888b-210b8e3f2258_characterizeImage_855136_10.stdout parsl.dataflow.dflow INFO: Standard error for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/85d0f654-ffcc-478a-888b-210b8e3f2258_characterizeImage_855136_10.stderr parsl.dataflow.dflow INFO: Standard output for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/226ab265-e240-485a-b95e-0d96c4b56442_isr_855136_10.stdout parsl.dataflow.dflow INFO: Standard error for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/226ab265-e240-485a-b95e-0d96c4b56442_isr_855136_10.stderr parsl.dataflow.dflow INFO: Task 1050 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91ffa259569aa3d551508f99528187e0 with result from task 1050 parsl.dataflow.dflow DEBUG: Task 1050 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1051 has memoization hash c5faaae8cd178d7e08533d52c49334cb parsl.dataflow.memoization INFO: Task 1051 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45542f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1051 try 0 launched on executor multi with executor id 666 parsl.dataflow.dflow INFO: Standard output for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/884c8170-c1f3-4702-9490-fc40690cd237_characterizeImage_855043_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/884c8170-c1f3-4702-9490-fc40690cd237_characterizeImage_855043_51.stderr parsl.dataflow.dflow INFO: Standard output for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/ae8f296e-6e5c-4a25-bf12-f511859f6e2e_isr_855043_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/ae8f296e-6e5c-4a25-bf12-f511859f6e2e_isr_855043_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:29 proc_lsst.multi:146 [INFO] found job 16912327 in provider escience_a40 proc_lsst.multi INFO: found job 16912327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 1002 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a7cb5e3ec1391f043124ce0655d85b9 with result from task 1002 parsl.dataflow.dflow DEBUG: Task 1002 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1003 has memoization hash 50ccfeca0b903a9ed991ae8a6767966f parsl.dataflow.memoization INFO: Task 1003 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454fbec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1003 try 0 launched on executor multi with executor id 667 parsl.dataflow.dflow INFO: Standard output for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/01b8cd61-948a-4d2c-8453-ebb5f5914407_characterizeImage_855136_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/01b8cd61-948a-4d2c-8453-ebb5f5914407_characterizeImage_855136_53.stderr parsl.dataflow.dflow INFO: Standard output for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/21a52822-db1d-4fe5-97eb-d946eb5d9ffb_isr_855136_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/21a52822-db1d-4fe5-97eb-d946eb5d9ffb_isr_855136_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:29 proc_lsst.multi:146 [INFO] found job 16912328 in provider escience_a40 proc_lsst.multi INFO: found job 16912328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 999 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10983e10acc270eeb45b42d413b816bb with result from task 999 parsl.dataflow.dflow DEBUG: Task 999 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1000 has memoization hash 0b47d24bc47b2c4681447e359c6436fb parsl.dataflow.memoization INFO: Task 1000 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454fbd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1000 try 0 launched on executor multi with executor id 668 parsl.dataflow.dflow INFO: Standard output for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/a96ef3dd-745b-4858-85cd-d464c276b70f_characterizeImage_855044_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/a96ef3dd-745b-4858-85cd-d464c276b70f_characterizeImage_855044_49.stderr parsl.dataflow.dflow INFO: Standard output for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/1b62f5ff-3adf-4c47-a705-f700666d7ce2_isr_855044_49.stdout parsl.dataflow.dflow INFO: Standard error for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/1b62f5ff-3adf-4c47-a705-f700666d7ce2_isr_855044_49.stderr parsl.dataflow.dflow INFO: Task 996 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c01da95e4186b7f002c404625526ce9 with result from task 996 parsl.dataflow.dflow DEBUG: Task 996 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 997 has memoization hash 9dac8b9774aec2248a551d3ce8ac0407 parsl.dataflow.memoization INFO: Task 997 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454fba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 997 try 0 launched on executor multi with executor id 669 parsl.dataflow.dflow INFO: Standard output for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/3b7f08a6-6510-4c13-85fb-a7ca4488e4dd_characterizeImage_855138_26.stdout parsl.dataflow.dflow INFO: Standard error for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/3b7f08a6-6510-4c13-85fb-a7ca4488e4dd_characterizeImage_855138_26.stderr parsl.dataflow.dflow INFO: Standard output for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/3606ee25-c31a-4769-9ec3-f8564ab413c8_isr_855138_26.stdout parsl.dataflow.dflow INFO: Standard error for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/3606ee25-c31a-4769-9ec3-f8564ab413c8_isr_855138_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:29 proc_lsst.multi:146 [INFO] found job 16912329 in provider escience_a40 proc_lsst.multi INFO: found job 16912329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 972 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e758e8904626e95a447c5750b76f20ab with result from task 972 parsl.dataflow.dflow DEBUG: Task 972 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 973 has memoization hash cb0974975f8132aa031b8640487a1f85 parsl.dataflow.memoization INFO: Task 973 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454f8f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 973 try 0 launched on executor multi with executor id 670 parsl.dataflow.dflow INFO: Standard output for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/75e3df4b-def6-457e-84bb-d962df526faa_characterizeImage_855137_10.stdout parsl.dataflow.dflow INFO: Standard error for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/75e3df4b-def6-457e-84bb-d962df526faa_characterizeImage_855137_10.stderr parsl.dataflow.dflow INFO: Standard output for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/ed34e911-41df-4374-a9a9-3477de67630c_isr_855137_10.stdout parsl.dataflow.dflow INFO: Standard error for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/ed34e911-41df-4374-a9a9-3477de67630c_isr_855137_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:30 proc_lsst.multi:146 [INFO] found job 16912330 in provider escience_a40 proc_lsst.multi INFO: found job 16912330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:30 proc_lsst.multi:146 [INFO] found job 16912331 in provider escience_a40 proc_lsst.multi INFO: found job 16912331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:31 proc_lsst.multi:146 [INFO] found job 16912332 in provider escience_a40 proc_lsst.multi INFO: found job 16912332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 1107 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1b9678b5dd284437c138057f12e3d32 with result from task 1107 parsl.dataflow.dflow DEBUG: Task 1107 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1108 has memoization hash 7e9046dab0c5bada3e9b75b83b91ce97 parsl.dataflow.memoization INFO: Task 1108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45392480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1108 try 0 launched on executor multi with executor id 671 parsl.dataflow.dflow INFO: Standard output for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/b9849438-8cfc-4435-a478-627529b331f8_characterizeImage_855138_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/b9849438-8cfc-4435-a478-627529b331f8_characterizeImage_855138_15.stderr parsl.dataflow.dflow INFO: Standard output for task 1107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/d96af26f-f987-4dc0-a77e-f77a676a61b6_isr_855138_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/d96af26f-f987-4dc0-a77e-f77a676a61b6_isr_855138_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:31 proc_lsst.multi:146 [INFO] found job 16912333 in provider escience_a40 proc_lsst.multi INFO: found job 16912333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 1008 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ab4c2435065fe37ffde973aa7d471ad with result from task 1008 parsl.dataflow.dflow DEBUG: Task 1008 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1009 has memoization hash c0b00c436ea77c889f4896474f83a5f8 parsl.dataflow.memoization INFO: Task 1009 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45540540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1009 try 0 launched on executor multi with executor id 672 parsl.dataflow.dflow INFO: Standard output for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/16c1c28d-d8d6-4d5d-a71c-c6089656cef7_characterizeImage_855043_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/16c1c28d-d8d6-4d5d-a71c-c6089656cef7_characterizeImage_855043_24.stderr parsl.dataflow.dflow INFO: Standard output for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/92cbee0e-8d38-420c-b7d6-784a5f5d6ec6_isr_855043_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/92cbee0e-8d38-420c-b7d6-784a5f5d6ec6_isr_855043_24.stderr parsl.dataflow.dflow INFO: Task 513 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65ecebbe71d6f27c3c780231de1ce243 with result from task 513 parsl.dataflow.dflow DEBUG: Task 513 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 514 has memoization hash b5f0f32698c228dbc6e3e6378049587e parsl.dataflow.memoization INFO: Task 514 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456cff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 514 try 0 launched on executor multi with executor id 673 parsl.dataflow.dflow INFO: Standard output for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/2e173f01-60d3-4f5d-9565-f0dd0bd841d2_characterizeImage_855043_41.stdout parsl.dataflow.dflow INFO: Standard error for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/2e173f01-60d3-4f5d-9565-f0dd0bd841d2_characterizeImage_855043_41.stderr parsl.dataflow.dflow INFO: Standard output for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/ad9def62-ef36-40e2-89da-49b6780f9c64_isr_855043_41.stdout parsl.dataflow.dflow INFO: Standard error for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/ad9def62-ef36-40e2-89da-49b6780f9c64_isr_855043_41.stderr parsl.dataflow.dflow INFO: Task 501 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c02d0d6c378c7a180e6d5bb9893f678 with result from task 501 parsl.dataflow.dflow DEBUG: Task 501 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 502 has memoization hash 2db874c7c92908b98d6010f3ac604a52 parsl.dataflow.memoization INFO: Task 502 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456ce520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 502 try 0 launched on executor multi with executor id 674 parsl.dataflow.dflow INFO: Standard output for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/d51ca415-1542-4450-8109-7d6fc1664e4c_characterizeImage_855042_13.stdout parsl.dataflow.dflow INFO: Standard error for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/d51ca415-1542-4450-8109-7d6fc1664e4c_characterizeImage_855042_13.stderr parsl.dataflow.dflow INFO: Standard output for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/ce520975-7be5-4fae-8d3d-e996d6e5fdef_isr_855042_13.stdout parsl.dataflow.dflow INFO: Standard error for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/ce520975-7be5-4fae-8d3d-e996d6e5fdef_isr_855042_13.stderr parsl.dataflow.dflow INFO: Task 504 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb44dffcc6c4979ac6b7f07f5f46b514 with result from task 504 parsl.dataflow.dflow DEBUG: Task 504 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 505 has memoization hash c2858b74e355a64f4a47de945ff44741 parsl.dataflow.memoization INFO: Task 505 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456cec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 505 try 0 launched on executor multi with executor id 675 parsl.dataflow.dflow INFO: Standard output for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/4a37ec0c-d957-4b53-a0ff-fbdc15050e92_characterizeImage_855042_15.stdout parsl.dataflow.dflow INFO: Standard error for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/4a37ec0c-d957-4b53-a0ff-fbdc15050e92_characterizeImage_855042_15.stderr parsl.dataflow.dflow INFO: Standard output for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/b752be77-cc5d-4438-92a1-b783ba5959a5_isr_855042_15.stdout parsl.dataflow.dflow INFO: Standard error for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/b752be77-cc5d-4438-92a1-b783ba5959a5_isr_855042_15.stderr parsl.dataflow.dflow INFO: Task 519 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af79bae9d64cdb231913ee3c91c40a72 with result from task 519 parsl.dataflow.dflow DEBUG: Task 519 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 520 has memoization hash e4f21d109f3cbf6049589a3efe06b866 parsl.dataflow.memoization INFO: Task 520 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456cc4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 520 try 0 launched on executor multi with executor id 676 parsl.dataflow.dflow INFO: Standard output for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/cb241c15-812f-4969-b025-c98ca1e7123d_characterizeImage_855137_41.stdout parsl.dataflow.dflow INFO: Standard error for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/cb241c15-812f-4969-b025-c98ca1e7123d_characterizeImage_855137_41.stderr parsl.dataflow.dflow INFO: Standard output for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/e4c88f10-6c79-47ce-98c1-88413c62fd06_isr_855137_41.stdout parsl.dataflow.dflow INFO: Standard error for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/e4c88f10-6c79-47ce-98c1-88413c62fd06_isr_855137_41.stderr parsl.dataflow.dflow INFO: Task 507 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82c627c7a08b63da0d2afe30d9e5e65a with result from task 507 parsl.dataflow.dflow DEBUG: Task 507 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 508 has memoization hash 6fe65ebd1b5b20ea9833c939cc1e2319 parsl.dataflow.memoization INFO: Task 508 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456ce840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 508 try 0 launched on executor multi with executor id 677 parsl.dataflow.dflow INFO: Standard output for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/a8bbf969-b430-43a0-80b5-b979329aede0_characterizeImage_855044_27.stdout parsl.dataflow.dflow INFO: Standard error for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/a8bbf969-b430-43a0-80b5-b979329aede0_characterizeImage_855044_27.stderr parsl.dataflow.dflow INFO: Standard output for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/e3012e20-3404-4d02-9529-18e033866b50_isr_855044_27.stdout parsl.dataflow.dflow INFO: Standard error for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/e3012e20-3404-4d02-9529-18e033866b50_isr_855044_27.stderr parsl.dataflow.dflow INFO: Task 498 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 970e3f6453d1bb0854e607c598d19854 with result from task 498 parsl.dataflow.dflow DEBUG: Task 498 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 499 has memoization hash ce2bbb321533d9fcde9b0862d820e545 parsl.dataflow.memoization INFO: Task 499 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456ce200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 499 try 0 launched on executor multi with executor id 678 parsl.dataflow.dflow INFO: Standard output for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/d08733f0-f3c6-4236-bd93-8d2e204c4db1_characterizeImage_855137_28.stdout parsl.dataflow.dflow INFO: Standard error for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/d08733f0-f3c6-4236-bd93-8d2e204c4db1_characterizeImage_855137_28.stderr parsl.dataflow.dflow INFO: Standard output for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/8d8eb2be-3878-4d8e-9c4c-d00d53cbaef6_isr_855137_28.stdout parsl.dataflow.dflow INFO: Standard error for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/8d8eb2be-3878-4d8e-9c4c-d00d53cbaef6_isr_855137_28.stderr parsl.dataflow.dflow INFO: Task 516 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9b9def846d1a0b1b467ee7a5f50e6a5 with result from task 516 parsl.dataflow.dflow DEBUG: Task 516 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 517 has memoization hash 4ff0a66e63eba8c89d77d92f9febc568 parsl.dataflow.memoization INFO: Task 517 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa457004a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 517 try 0 launched on executor multi with executor id 679 parsl.dataflow.dflow INFO: Standard output for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/1da18e3e-dea3-4a53-8e8d-8334bc0d616e_characterizeImage_855137_17.stdout parsl.dataflow.dflow INFO: Standard error for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/1da18e3e-dea3-4a53-8e8d-8334bc0d616e_characterizeImage_855137_17.stderr parsl.dataflow.dflow INFO: Standard output for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/910e0660-ef1e-4b11-8324-de22875c6300_isr_855137_17.stdout parsl.dataflow.dflow INFO: Standard error for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/910e0660-ef1e-4b11-8324-de22875c6300_isr_855137_17.stderr parsl.dataflow.dflow INFO: Task 510 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5bdba4a1b0ded83afa799fa14d2da3c with result from task 510 parsl.dataflow.dflow DEBUG: Task 510 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 511 has memoization hash c403083d8a8dbd4b21e8b4ef2bcc0bac parsl.dataflow.memoization INFO: Task 511 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456cfe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 511 try 0 launched on executor multi with executor id 680 parsl.dataflow.dflow INFO: Standard output for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/11432293-3e64-4f5a-ae6a-0051f04af7ef_characterizeImage_855042_44.stdout parsl.dataflow.dflow INFO: Standard error for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/11432293-3e64-4f5a-ae6a-0051f04af7ef_characterizeImage_855042_44.stderr parsl.dataflow.dflow INFO: Standard output for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/4aced42b-8200-4ec1-957d-425ba34f1d81_isr_855042_44.stdout parsl.dataflow.dflow INFO: Standard error for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/4aced42b-8200-4ec1-957d-425ba34f1d81_isr_855042_44.stderr parsl.dataflow.dflow INFO: Task 1065 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67ab182f690618bf3f50b51e3d588c42 with result from task 1065 parsl.dataflow.dflow DEBUG: Task 1065 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1066 has memoization hash 30ee99cea6ac39d58b868193ed6a56fc parsl.dataflow.memoization INFO: Task 1066 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45393240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1066 try 0 launched on executor multi with executor id 681 parsl.dataflow.dflow INFO: Standard output for task 1066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/104e703f-17b1-472c-8d4f-2610345dee80_characterizeImage_855138_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/104e703f-17b1-472c-8d4f-2610345dee80_characterizeImage_855138_19.stderr parsl.dataflow.dflow INFO: Standard output for task 1065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/4ee468d3-09cf-44b8-9218-d745d75011cc_isr_855138_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/4ee468d3-09cf-44b8-9218-d745d75011cc_isr_855138_19.stderr parsl.dataflow.dflow INFO: Task 522 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb197d534e90cca838ac423836bb9def with result from task 522 parsl.dataflow.dflow DEBUG: Task 522 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 523 has memoization hash 591e05ef05082a4ce17e056b437c2d32 parsl.dataflow.memoization INFO: Task 523 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa457005e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 523 try 0 launched on executor multi with executor id 682 parsl.dataflow.dflow INFO: Standard output for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/dd92eb2a-0ec0-4c4c-8b1d-726be4addc7d_characterizeImage_855137_39.stdout parsl.dataflow.dflow INFO: Standard error for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/dd92eb2a-0ec0-4c4c-8b1d-726be4addc7d_characterizeImage_855137_39.stderr parsl.dataflow.dflow INFO: Standard output for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/300a5f14-ac72-4817-8114-77ec84973af4_isr_855137_39.stdout parsl.dataflow.dflow INFO: Standard error for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/300a5f14-ac72-4817-8114-77ec84973af4_isr_855137_39.stderr parsl.dataflow.dflow INFO: Task 540 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d4777dcb2afc7f7724a13b68ae6523a with result from task 540 parsl.dataflow.dflow DEBUG: Task 540 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 541 has memoization hash dbe37818638973db44c0eb5cc6c0b07d parsl.dataflow.memoization INFO: Task 541 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45701440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 541 try 0 launched on executor multi with executor id 683 parsl.dataflow.dflow INFO: Standard output for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/2b8da6f1-b46e-4f6e-a20e-1fbfdfbbf610_characterizeImage_855044_20.stdout parsl.dataflow.dflow INFO: Standard error for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/2b8da6f1-b46e-4f6e-a20e-1fbfdfbbf610_characterizeImage_855044_20.stderr parsl.dataflow.dflow INFO: Standard output for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/45b5e960-377b-4f5d-bd3c-2a7f17c220d3_isr_855044_20.stdout parsl.dataflow.dflow INFO: Standard error for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/45b5e960-377b-4f5d-bd3c-2a7f17c220d3_isr_855044_20.stderr parsl.dataflow.dflow INFO: Task 531 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ba9c617e382ce3da0d2df60698f4087 with result from task 531 parsl.dataflow.dflow DEBUG: Task 531 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 532 has memoization hash 17dfec860bc0662f83dceb23492de9e0 parsl.dataflow.memoization INFO: Task 532 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45703920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 532 try 0 launched on executor multi with executor id 684 parsl.dataflow.dflow INFO: Standard output for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/4cb5ddd4-8f4d-4a37-a6c2-6ba6b178ffdc_characterizeImage_855042_32.stdout parsl.dataflow.dflow INFO: Standard error for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/4cb5ddd4-8f4d-4a37-a6c2-6ba6b178ffdc_characterizeImage_855042_32.stderr parsl.dataflow.dflow INFO: Standard output for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/dc1bcaac-aeb8-4c8a-bccf-fe1aafe62f41_isr_855042_32.stdout parsl.dataflow.dflow INFO: Standard error for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/dc1bcaac-aeb8-4c8a-bccf-fe1aafe62f41_isr_855042_32.stderr parsl.dataflow.dflow INFO: Task 537 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3fc68b7b32af5d5a3d6fe78d466fb170 with result from task 537 parsl.dataflow.dflow DEBUG: Task 537 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 538 has memoization hash 466e02d665a5de1c81b29ba06a307cd5 parsl.dataflow.memoization INFO: Task 538 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa457011c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 538 try 0 launched on executor multi with executor id 685 parsl.dataflow.dflow INFO: Standard output for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/9376f528-6e14-4f1a-9383-16a250ef90a3_characterizeImage_855043_10.stdout parsl.dataflow.dflow INFO: Standard error for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/9376f528-6e14-4f1a-9383-16a250ef90a3_characterizeImage_855043_10.stderr parsl.dataflow.dflow INFO: Standard output for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/529d2866-60c7-4197-a72e-ce3bcb5c2f00_isr_855043_10.stdout parsl.dataflow.dflow INFO: Standard error for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/529d2866-60c7-4197-a72e-ce3bcb5c2f00_isr_855043_10.stderr parsl.dataflow.dflow INFO: Task 525 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3726acaa1bb5d5a64f9a03bf3869b914 with result from task 525 parsl.dataflow.dflow DEBUG: Task 525 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 526 has memoization hash 6b0782db124ca6ea432056505545aa2d parsl.dataflow.memoization INFO: Task 526 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45700c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 526 try 0 launched on executor multi with executor id 686 parsl.dataflow.dflow INFO: Standard output for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/245d99d9-5f43-4cb3-a15a-ba0771ee70fd_characterizeImage_855042_26.stdout parsl.dataflow.dflow INFO: Standard error for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/245d99d9-5f43-4cb3-a15a-ba0771ee70fd_characterizeImage_855042_26.stderr parsl.dataflow.dflow INFO: Standard output for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/ce4acd6b-7b43-407e-bd08-6d9fd331679b_isr_855042_26.stdout parsl.dataflow.dflow INFO: Standard error for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/ce4acd6b-7b43-407e-bd08-6d9fd331679b_isr_855042_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:31 proc_lsst.multi:146 [INFO] found job 16912334 in provider escience_a40 proc_lsst.multi INFO: found job 16912334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 1098 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1d944ad77d97029a430694d7226d9b2 with result from task 1098 parsl.dataflow.dflow DEBUG: Task 1098 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1099 has memoization hash a390ed2e4656a37563152dc783c02716 parsl.dataflow.memoization INFO: Task 1099 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa453919e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1099 try 0 launched on executor multi with executor id 687 parsl.dataflow.dflow INFO: Standard output for task 1099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/e5d5efe8-cbb8-44fd-b5bc-1c7849807054_characterizeImage_855042_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/e5d5efe8-cbb8-44fd-b5bc-1c7849807054_characterizeImage_855042_46.stderr parsl.dataflow.dflow INFO: Standard output for task 1098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/cec5c57e-142c-4c54-9461-bed07c9cbcfb_isr_855042_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/cec5c57e-142c-4c54-9461-bed07c9cbcfb_isr_855042_46.stderr parsl.dataflow.dflow INFO: Task 1017 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4e536a041c9ec4c4b4ec338d31e3267 with result from task 1017 parsl.dataflow.dflow DEBUG: Task 1017 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1018 has memoization hash 1005e0d1f908c1516755ed5e2d8b799a parsl.dataflow.memoization INFO: Task 1018 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45540d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1018 try 0 launched on executor multi with executor id 688 parsl.dataflow.dflow INFO: Standard output for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/8dbb4b2f-d5bb-4f3f-a084-60111c0464a2_characterizeImage_855137_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/8dbb4b2f-d5bb-4f3f-a084-60111c0464a2_characterizeImage_855137_24.stderr parsl.dataflow.dflow INFO: Standard output for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/79c4b19a-a445-4378-8983-aab03df78370_isr_855137_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/79c4b19a-a445-4378-8983-aab03df78370_isr_855137_24.stderr parsl.dataflow.dflow INFO: Task 1041 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63550a08f0af41f3cf34c9fb19b52bae with result from task 1041 parsl.dataflow.dflow DEBUG: Task 1041 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1042 has memoization hash 412793f7a6f58799ba29a6c1be86a6a3 parsl.dataflow.memoization INFO: Task 1042 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45542660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1042 try 0 launched on executor multi with executor id 689 parsl.dataflow.dflow INFO: Standard output for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/34e5a714-87e6-4c51-8cc5-5b139042af69_characterizeImage_855136_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/34e5a714-87e6-4c51-8cc5-5b139042af69_characterizeImage_855136_16.stderr parsl.dataflow.dflow INFO: Standard output for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/15526cc5-18e7-47cb-bc75-3733d361b914_isr_855136_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/15526cc5-18e7-47cb-bc75-3733d361b914_isr_855136_16.stderr parsl.dataflow.dflow INFO: Task 1005 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7008df90a3226ec351bf10835807ba25 with result from task 1005 parsl.dataflow.dflow DEBUG: Task 1005 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1006 has memoization hash 9adb080bb264f3242c46a2fe8e58ddc5 parsl.dataflow.memoization INFO: Task 1006 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4549bc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1006 try 0 launched on executor multi with executor id 690 parsl.dataflow.dflow INFO: Standard output for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/dca720a9-7944-48cd-90ec-5118c3b99bae_characterizeImage_855042_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/dca720a9-7944-48cd-90ec-5118c3b99bae_characterizeImage_855042_23.stderr parsl.dataflow.dflow INFO: Standard output for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/c6bd814d-cbd5-4236-858a-f5a37cd9fd5a_isr_855042_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/c6bd814d-cbd5-4236-858a-f5a37cd9fd5a_isr_855042_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:32 proc_lsst.multi:146 [INFO] found job 16912335 in provider escience_a40 proc_lsst.multi INFO: found job 16912335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 1077 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c47e6187610fd143ce9b044cefb8653 with result from task 1077 parsl.dataflow.dflow DEBUG: Task 1077 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1078 has memoization hash b6e2433c77050931dfa29a14e897c438 parsl.dataflow.memoization INFO: Task 1078 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa453902c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1078 try 0 launched on executor multi with executor id 691 parsl.dataflow.dflow INFO: Standard output for task 1078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/9d1f5f35-5552-464f-b077-593393c3f77a_characterizeImage_855138_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/9d1f5f35-5552-464f-b077-593393c3f77a_characterizeImage_855138_28.stderr parsl.dataflow.dflow INFO: Standard output for task 1077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/91d6f87d-593b-4ae3-804b-4aa795923863_isr_855138_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/91d6f87d-593b-4ae3-804b-4aa795923863_isr_855138_28.stderr parsl.dataflow.dflow INFO: Task 984 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ed533cda13dde14e8b45a8c6460c26d with result from task 984 parsl.dataflow.dflow DEBUG: Task 984 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 985 has memoization hash 429b164cda52711389ced1c86627f3a7 parsl.dataflow.memoization INFO: Task 985 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454f9c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 985 try 0 launched on executor multi with executor id 692 parsl.dataflow.dflow INFO: Standard output for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/9db6b781-cf39-456a-974f-f9f6baa137fa_characterizeImage_855138_20.stdout parsl.dataflow.dflow INFO: Standard error for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/9db6b781-cf39-456a-974f-f9f6baa137fa_characterizeImage_855138_20.stderr parsl.dataflow.dflow INFO: Standard output for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/08721a85-9aab-4bb9-9ddf-b9b2b5ad213a_isr_855138_20.stdout parsl.dataflow.dflow INFO: Standard error for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/08721a85-9aab-4bb9-9ddf-b9b2b5ad213a_isr_855138_20.stderr parsl.dataflow.dflow INFO: Task 1071 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5f877bd45bc611dade83ac4c4dbddcd with result from task 1071 parsl.dataflow.dflow DEBUG: Task 1071 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1072 has memoization hash 1662213a7df9f9b7197b055e58d39580 parsl.dataflow.memoization INFO: Task 1072 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45392c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1072 try 0 launched on executor multi with executor id 693 parsl.dataflow.dflow INFO: Standard output for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/0275787f-ea3a-4b32-909a-45e4bbbcfd6e_characterizeImage_855136_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/0275787f-ea3a-4b32-909a-45e4bbbcfd6e_characterizeImage_855136_24.stderr parsl.dataflow.dflow INFO: Standard output for task 1071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/e4b007e3-17ac-44be-ab15-10976b0d0977_isr_855136_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/e4b007e3-17ac-44be-ab15-10976b0d0977_isr_855136_24.stderr parsl.dataflow.dflow INFO: Task 1053 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae28f44aec13179e12bcb17a86e1b661 with result from task 1053 parsl.dataflow.dflow DEBUG: Task 1053 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1054 has memoization hash e638114a4a6f25400844d23175a5d083 parsl.dataflow.memoization INFO: Task 1054 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45542c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1054 try 0 launched on executor multi with executor id 694 parsl.dataflow.dflow INFO: Standard output for task 1054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/b9894a9c-e2bc-45cd-91d0-367729e4463f_characterizeImage_855043_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/b9894a9c-e2bc-45cd-91d0-367729e4463f_characterizeImage_855043_9.stderr parsl.dataflow.dflow INFO: Standard output for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/2236bb40-2430-4463-82e8-66c9c0810c68_isr_855043_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/2236bb40-2430-4463-82e8-66c9c0810c68_isr_855043_9.stderr parsl.dataflow.dflow INFO: Task 1023 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c6e01a79ef47cea1257b6552bbafe7f with result from task 1023 parsl.dataflow.dflow DEBUG: Task 1023 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1024 has memoization hash 3f18dc60954ca83dafa2dc533ee0e136 parsl.dataflow.memoization INFO: Task 1024 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455411c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1024 try 0 launched on executor multi with executor id 695 parsl.dataflow.dflow INFO: Standard output for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/5882e7dd-b851-42c7-87c2-0deaa58ff76f_characterizeImage_855043_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/5882e7dd-b851-42c7-87c2-0deaa58ff76f_characterizeImage_855043_11.stderr parsl.dataflow.dflow INFO: Standard output for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/936fe576-4d39-4579-b7cd-e9379d29836a_isr_855043_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/936fe576-4d39-4579-b7cd-e9379d29836a_isr_855043_11.stderr parsl.dataflow.dflow INFO: Task 543 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf4ff54f0fcc73e45ce05e0e1d76c24c with result from task 543 parsl.dataflow.dflow DEBUG: Task 543 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 544 has memoization hash c30519cbd1108bdea301a66c6c0119eb parsl.dataflow.memoization INFO: Task 544 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45701760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 544 try 0 launched on executor multi with executor id 696 parsl.dataflow.dflow INFO: Standard output for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/62d5cdfa-e4a5-4dc8-affc-7b7288eb1429_characterizeImage_855138_18.stdout parsl.dataflow.dflow INFO: Standard error for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/62d5cdfa-e4a5-4dc8-affc-7b7288eb1429_characterizeImage_855138_18.stderr parsl.dataflow.dflow INFO: Standard output for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/e4076bcb-615d-43e1-9dd1-69b7a2af3c8d_isr_855138_18.stdout parsl.dataflow.dflow INFO: Standard error for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/e4076bcb-615d-43e1-9dd1-69b7a2af3c8d_isr_855138_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:32 proc_lsst.multi:146 [INFO] found job 16912336 in provider escience_a40 proc_lsst.multi INFO: found job 16912336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 1104 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 248b54f520dbcca6fbf5a2c90c743c3f with result from task 1104 parsl.dataflow.dflow DEBUG: Task 1104 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1105 has memoization hash 91498483b312b628ffa8b3da897ae412 parsl.dataflow.memoization INFO: Task 1105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45392020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1105 try 0 launched on executor multi with executor id 697 parsl.dataflow.dflow INFO: Standard output for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/2b91ebb6-bb74-4077-8d34-fa114cf95594_characterizeImage_855138_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/2b91ebb6-bb74-4077-8d34-fa114cf95594_characterizeImage_855138_23.stderr parsl.dataflow.dflow INFO: Standard output for task 1104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/d941a126-9b8f-4e8b-bc08-2617a88bab4d_isr_855138_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/d941a126-9b8f-4e8b-bc08-2617a88bab4d_isr_855138_23.stderr parsl.dataflow.dflow INFO: Task 528 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d275d23c5c1cad760a3a20a65aa04adf with result from task 528 parsl.dataflow.dflow DEBUG: Task 528 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 529 has memoization hash 8552d79791080e65f8b20fa93c620c12 parsl.dataflow.memoization INFO: Task 529 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45703c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 529 try 0 launched on executor multi with executor id 698 parsl.dataflow.dflow INFO: Standard output for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/389e2677-d604-47c9-976f-31cb1ab0bb10_characterizeImage_855044_38.stdout parsl.dataflow.dflow INFO: Standard error for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/389e2677-d604-47c9-976f-31cb1ab0bb10_characterizeImage_855044_38.stderr parsl.dataflow.dflow INFO: Standard output for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/7a94c3c8-f91b-4c5c-b69f-c5a932a0f8c6_isr_855044_38.stdout parsl.dataflow.dflow INFO: Standard error for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/7a94c3c8-f91b-4c5c-b69f-c5a932a0f8c6_isr_855044_38.stderr parsl.dataflow.dflow INFO: Task 1059 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e5c1dfab2b86f2d0eb5bf13aba0106c with result from task 1059 parsl.dataflow.dflow DEBUG: Task 1059 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1060 has memoization hash d226509f496aa2e7942e9e62821d8534 parsl.dataflow.memoization INFO: Task 1060 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45543c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1060 try 0 launched on executor multi with executor id 699 parsl.dataflow.dflow INFO: Standard output for task 1060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/142c9c74-aff9-4a43-803d-6c967b1c492d_characterizeImage_855137_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/142c9c74-aff9-4a43-803d-6c967b1c492d_characterizeImage_855137_11.stderr parsl.dataflow.dflow INFO: Standard output for task 1059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/57991402-97d2-4cd0-bf5e-22f4c97d5903_isr_855137_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/57991402-97d2-4cd0-bf5e-22f4c97d5903_isr_855137_11.stderr parsl.dataflow.dflow INFO: Task 1089 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 541c95755d58dfec5e805be55203df78 with result from task 1089 parsl.dataflow.dflow DEBUG: Task 1089 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1090 has memoization hash db02db5edd399c080aa57d093f7f93b5 parsl.dataflow.memoization INFO: Task 1090 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45391080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1090 try 0 launched on executor multi with executor id 700 parsl.dataflow.dflow INFO: Standard output for task 1090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/9ee49d06-ae20-45e4-8a4e-a348c8876e18_characterizeImage_855136_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/9ee49d06-ae20-45e4-8a4e-a348c8876e18_characterizeImage_855136_17.stderr parsl.dataflow.dflow INFO: Standard output for task 1089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/1bd78a6a-276c-4092-95f1-9e4de9e21321_isr_855136_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/1bd78a6a-276c-4092-95f1-9e4de9e21321_isr_855136_17.stderr parsl.dataflow.dflow INFO: Task 1062 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d8b68465a3c0d4f23a0d025deee951e with result from task 1062 parsl.dataflow.dflow DEBUG: Task 1062 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1063 has memoization hash 30e9aa5d672cdf61120c8f55d1f55ed5 parsl.dataflow.memoization INFO: Task 1063 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45543d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1063 try 0 launched on executor multi with executor id 701 parsl.dataflow.dflow INFO: Standard output for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/17b5038d-27ad-4408-86e8-c82832441379_characterizeImage_855044_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/17b5038d-27ad-4408-86e8-c82832441379_characterizeImage_855044_19.stderr parsl.dataflow.dflow INFO: Standard output for task 1062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/3a598482-7f66-4a83-b463-4242aaa61377_isr_855044_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/3a598482-7f66-4a83-b463-4242aaa61377_isr_855044_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:32 proc_lsst.multi:146 [INFO] found job 16912337 in provider escience_a40 proc_lsst.multi INFO: found job 16912337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 1086 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 28c051a01cb4108b0e6f91f57122ce56 with result from task 1086 parsl.dataflow.dflow DEBUG: Task 1086 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1087 has memoization hash c7e0f0b141e83fa911f3bdc84cf68d42 parsl.dataflow.memoization INFO: Task 1087 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45390cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1087 try 0 launched on executor multi with executor id 702 parsl.dataflow.dflow INFO: Standard output for task 1087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/4602f08e-15b0-4ced-abf2-7a93e0005b5a_characterizeImage_855136_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/4602f08e-15b0-4ced-abf2-7a93e0005b5a_characterizeImage_855136_56.stderr parsl.dataflow.dflow INFO: Standard output for task 1086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/952abdf3-8f1b-4297-8189-6144b865bad8_isr_855136_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/952abdf3-8f1b-4297-8189-6144b865bad8_isr_855136_56.stderr parsl.dataflow.dflow INFO: Task 1011 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 74f5128366cffe96b15e166e9feb7955 with result from task 1011 parsl.dataflow.dflow DEBUG: Task 1011 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1012 has memoization hash 438ed5ddb54c0c740485d7cb5620a9b8 parsl.dataflow.memoization INFO: Task 1012 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45543880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1012 try 0 launched on executor multi with executor id 703 parsl.dataflow.dflow INFO: Standard output for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/8c313872-7956-4a64-868f-00269892358f_characterizeImage_855138_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/8c313872-7956-4a64-868f-00269892358f_characterizeImage_855138_59.stderr parsl.dataflow.dflow INFO: Standard output for task 1011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/fe4c95c4-a670-4184-8ac4-52fa10c6a2ee_isr_855138_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/fe4c95c4-a670-4184-8ac4-52fa10c6a2ee_isr_855138_59.stderr parsl.dataflow.dflow INFO: Task 1020 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0649aefc3472bb6e31d36fa43f825bd3 with result from task 1020 parsl.dataflow.dflow DEBUG: Task 1020 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1021 has memoization hash 64cf7749d327fa5b317a0848009982d3 parsl.dataflow.memoization INFO: Task 1021 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45541080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1021 try 0 launched on executor multi with executor id 704 parsl.dataflow.dflow INFO: Standard output for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/e856011a-bbee-491b-b894-ea7c7f837ce6_characterizeImage_855042_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/e856011a-bbee-491b-b894-ea7c7f837ce6_characterizeImage_855042_60.stderr parsl.dataflow.dflow INFO: Standard output for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/28f2e87d-415d-46c1-b263-c9cad74eb93f_isr_855042_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/28f2e87d-415d-46c1-b263-c9cad74eb93f_isr_855042_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:33 proc_lsst.multi:146 [INFO] found job 16912338 in provider checkpoint proc_lsst.multi INFO: found job 16912338 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375' parsl.dataflow.dflow INFO: Task 534 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2472f23f2d09bac1191f9ec52387c664 with result from task 534 parsl.dataflow.dflow DEBUG: Task 534 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 535 has memoization hash bf4605cef489954be5580490b2014b59 parsl.dataflow.memoization INFO: Task 535 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45703600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 535 try 0 launched on executor multi with executor id 705 parsl.dataflow.dflow INFO: Standard output for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/6669a619-ccd6-4d0b-a6be-d89d794f9d2d_characterizeImage_855042_7.stdout parsl.dataflow.dflow INFO: Standard error for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/6669a619-ccd6-4d0b-a6be-d89d794f9d2d_characterizeImage_855042_7.stderr parsl.dataflow.dflow INFO: Standard output for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/37fe96d8-e571-4da3-8893-5362b056eea5_isr_855042_7.stdout parsl.dataflow.dflow INFO: Standard error for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/37fe96d8-e571-4da3-8893-5362b056eea5_isr_855042_7.stderr parsl.dataflow.dflow INFO: Task 1014 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e3ad07164a07c4dbea96ead8cd398d0 with result from task 1014 parsl.dataflow.dflow DEBUG: Task 1014 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1015 has memoization hash 09598be12c1d3c26e5c92fa26bc52474 parsl.dataflow.memoization INFO: Task 1015 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45540ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1015 try 0 launched on executor multi with executor id 706 parsl.dataflow.dflow INFO: Standard output for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/81857641-7f3a-421c-994d-05279ac99483_characterizeImage_855138_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/81857641-7f3a-421c-994d-05279ac99483_characterizeImage_855138_52.stderr parsl.dataflow.dflow INFO: Standard output for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/c0bb59ce-6e0d-4f4a-83a0-09d65537bccc_isr_855138_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/c0bb59ce-6e0d-4f4a-83a0-09d65537bccc_isr_855138_52.stderr parsl.dataflow.dflow INFO: Task 1080 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d03be96244256fd6dc38935cd2a8e58e with result from task 1080 parsl.providers.slurm.slurm DEBUG: squeue returned 16912363 PD 16912364 PD 16912365 PD 16912366 PD 16912367 PD 16912368 PD 16912369 PD 16912370 PD 16912371 PD 16912372 PD 16912373 PD 16912374 PD 16912375 PD 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:33 proc_lsst.multi:146 [INFO] found job 16912339 in provider checkpoint proc_lsst.multi INFO: found job 16912339 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375' parsl.dataflow.dflow DEBUG: Task 1080 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1081 has memoization hash e07cf5391f82f96308bed92916e4bb0a parsl.dataflow.memoization INFO: Task 1081 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45390720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1081 try 0 launched on executor multi with executor id 707 parsl.dataflow.dflow INFO: Standard output for task 1081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/dead2eb3-23ef-4210-80df-eef151c41d4c_characterizeImage_855043_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/dead2eb3-23ef-4210-80df-eef151c41d4c_characterizeImage_855043_22.stderr parsl.dataflow.dflow INFO: Standard output for task 1080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/0906a5c5-e998-4970-a860-b9d5ef14a426_isr_855043_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/0906a5c5-e998-4970-a860-b9d5ef14a426_isr_855043_22.stderr parsl.dataflow.dflow INFO: Task 1029 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 924c3071a231e25bb03ccb71075ceabd with result from task 1029 parsl.dataflow.dflow DEBUG: Task 1029 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1030 has memoization hash 33a4f2ab1c376f491a52a029327765e2 parsl.dataflow.memoization INFO: Task 1030 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45541940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1030 try 0 launched on executor multi with executor id 708 parsl.dataflow.dflow INFO: Standard output for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/19ebfe5c-9f78-4413-a9b8-f111de721222_characterizeImage_855042_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/19ebfe5c-9f78-4413-a9b8-f111de721222_characterizeImage_855042_40.stderr parsl.dataflow.dflow INFO: Standard output for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/1fd4a95f-e0a5-47e2-8b72-c74b08ac9845_isr_855042_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/1fd4a95f-e0a5-47e2-8b72-c74b08ac9845_isr_855042_40.stderr parsl.dataflow.dflow INFO: Task 1026 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2dc81e1e980550f8abdecfcf44b45c42 with result from task 1026 parsl.dataflow.dflow DEBUG: Task 1026 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1027 has memoization hash c386a321548e15810448d85624868210 parsl.dataflow.memoization INFO: Task 1027 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455414e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1027 try 0 launched on executor multi with executor id 709 parsl.dataflow.dflow INFO: Standard output for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/03b39fdf-fa97-4709-ad6b-24d8feb4e95e_characterizeImage_855042_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/03b39fdf-fa97-4709-ad6b-24d8feb4e95e_characterizeImage_855042_56.stderr parsl.dataflow.dflow INFO: Standard output for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/fadbdff8-8e84-4a26-b7e3-7045401ee5e8_isr_855042_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/fadbdff8-8e84-4a26-b7e3-7045401ee5e8_isr_855042_56.stderr parsl.dataflow.dflow INFO: Task 1032 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb252179bc75a74ed359abb0c6860f9d with result from task 1032 parsl.dataflow.dflow DEBUG: Task 1032 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1033 has memoization hash 202b930ffca9ee2d7015182856d3f74c parsl.dataflow.memoization INFO: Task 1033 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45541c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1033 try 0 launched on executor multi with executor id 710 parsl.dataflow.dflow INFO: Standard output for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/e8148b0d-7a16-4dba-8b12-b7f19f4d28b7_characterizeImage_855043_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/e8148b0d-7a16-4dba-8b12-b7f19f4d28b7_characterizeImage_855043_38.stderr parsl.dataflow.dflow INFO: Standard output for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/0ef5d4d9-bfd3-4dac-8a75-89f82462f9fc_isr_855043_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/0ef5d4d9-bfd3-4dac-8a75-89f82462f9fc_isr_855043_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912363 PD 16912364 PD 16912365 PD 16912366 PD 16912367 PD 16912368 PD 16912369 PD 16912370 PD 16912371 PD 16912372 PD 16912373 PD 16912374 PD 16912375 PD 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:33 proc_lsst.multi:146 [INFO] found job 16912340 in provider checkpoint proc_lsst.multi INFO: found job 16912340 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375' parsl.dataflow.dflow INFO: Task 1047 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 40949c8d69ba4eed70ae69167c699778 with result from task 1047 parsl.dataflow.dflow DEBUG: Task 1047 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1048 has memoization hash 4b98af4b47132c029fc5f14452a56943 parsl.dataflow.memoization INFO: Task 1048 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45543240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1048 try 0 launched on executor multi with executor id 711 parsl.dataflow.dflow INFO: Standard output for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/bf774540-4587-4429-871e-307b3e84f4b3_characterizeImage_855137_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/bf774540-4587-4429-871e-307b3e84f4b3_characterizeImage_855137_9.stderr parsl.dataflow.dflow INFO: Standard output for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/4107f3d6-babe-4a94-8752-49db50fb6ace_isr_855137_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/4107f3d6-babe-4a94-8752-49db50fb6ace_isr_855137_9.stderr parsl.dataflow.dflow INFO: Task 1038 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5190eea96c4199376e0da29d9d4d2f00 with result from task 1038 parsl.dataflow.dflow DEBUG: Task 1038 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1039 has memoization hash e1ca11e370a19e38dbaa3022b316e021 parsl.dataflow.memoization INFO: Task 1039 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455423e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1039 try 0 launched on executor multi with executor id 712 parsl.dataflow.dflow INFO: Standard output for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/226247c2-de41-4cd1-8d17-aec7cc613876_characterizeImage_855138_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/226247c2-de41-4cd1-8d17-aec7cc613876_characterizeImage_855138_61.stderr parsl.dataflow.dflow INFO: Standard output for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/9eb3c489-c634-4089-ac86-dd19d00e6af2_isr_855138_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/9eb3c489-c634-4089-ac86-dd19d00e6af2_isr_855138_61.stderr parsl.dataflow.dflow INFO: Task 1092 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5b732ba1939bd5d580d301824833180 with result from task 1092 parsl.dataflow.dflow DEBUG: Task 1092 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1093 has memoization hash 32fb35da4f500e649101280e028f260c parsl.dataflow.memoization INFO: Task 1093 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa453913a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1093 try 0 launched on executor multi with executor id 713 parsl.dataflow.dflow INFO: Standard output for task 1093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/a2168549-6764-437e-9864-0b7049889c86_characterizeImage_855043_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/a2168549-6764-437e-9864-0b7049889c86_characterizeImage_855043_23.stderr parsl.dataflow.dflow INFO: Standard output for task 1092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/eca75e3c-41c8-4245-a78c-ea0a7663fd59_isr_855043_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/eca75e3c-41c8-4245-a78c-ea0a7663fd59_isr_855043_23.stderr parsl.dataflow.dflow INFO: Task 978 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b08b0b20ef07311bec6f792b9cc9e37c with result from task 978 parsl.dataflow.dflow DEBUG: Task 978 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 979 has memoization hash 19c4d6cfdedaa3325757b968648587d0 parsl.dataflow.memoization INFO: Task 979 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454f94e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 979 try 0 launched on executor multi with executor id 714 parsl.dataflow.dflow INFO: Standard output for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/18b786a3-284b-4df4-a757-01fc90707683_characterizeImage_855138_11.stdout parsl.dataflow.dflow INFO: Standard error for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/18b786a3-284b-4df4-a757-01fc90707683_characterizeImage_855138_11.stderr parsl.dataflow.dflow INFO: Standard output for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/0fa03805-a20e-4086-bdda-e9cef76c9ac8_isr_855138_11.stdout parsl.dataflow.dflow INFO: Standard error for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/0fa03805-a20e-4086-bdda-e9cef76c9ac8_isr_855138_11.stderr parsl.dataflow.dflow INFO: Task 1044 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7602e7b5b413f7e1e6da832650f7c023 with result from task 1044 parsl.dataflow.dflow DEBUG: Task 1044 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1045 has memoization hash 35ea2bf0db1d80e5cdac3c6350ea1a91 parsl.dataflow.memoization INFO: Task 1045 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45543560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1045 try 0 launched on executor multi with executor id 715 parsl.dataflow.dflow INFO: Standard output for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/b6a8fa87-aa73-4766-ab7f-2140139ee391_characterizeImage_855044_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/b6a8fa87-aa73-4766-ab7f-2140139ee391_characterizeImage_855044_54.stderr parsl.dataflow.dflow INFO: Standard output for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/09f2bc0c-f482-48ed-b70e-d3837454400e_isr_855044_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/09f2bc0c-f482-48ed-b70e-d3837454400e_isr_855044_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912363 PD 16912364 PD 16912365 PD 16912366 PD 16912367 PD 16912368 PD 16912369 PD 16912370 PD 16912371 PD 16912372 PD 16912373 PD 16912374 PD 16912375 PD 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:34 proc_lsst.multi:146 [INFO] found job 16912341 in provider checkpoint proc_lsst.multi INFO: found job 16912341 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375' parsl.providers.slurm.slurm DEBUG: squeue returned 16912363 PD 16912364 PD 16912365 PD 16912366 PD 16912367 PD 16912368 PD 16912369 PD 16912370 PD 16912371 PD 16912372 PD 16912373 PD 16912374 PD 16912375 PD 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:34 proc_lsst.multi:146 [INFO] found job 16912342 in provider checkpoint proc_lsst.multi INFO: found job 16912342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375' parsl.providers.slurm.slurm DEBUG: squeue returned 16912363 PD 16912364 PD 16912365 PD 16912366 PD 16912367 PD 16912368 PD 16912369 PD 16912370 PD 16912371 PD 16912372 PD 16912373 PD 16912374 PD 16912375 PD 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:34 proc_lsst.multi:146 [INFO] found job 16912343 in provider checkpoint proc_lsst.multi INFO: found job 16912343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375' parsl.dataflow.dflow INFO: Task 1035 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 622b2b62eaf673638731604ebb69e3c7 with result from task 1035 parsl.dataflow.dflow DEBUG: Task 1035 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1036 has memoization hash 7263450d2fcdbe2ebcc04e732a2ddb3f parsl.dataflow.memoization INFO: Task 1036 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45541ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1036 try 0 launched on executor multi with executor id 716 parsl.dataflow.dflow INFO: Standard output for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/0c80d029-2691-415f-aec7-435068ce2978_characterizeImage_855136_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/0c80d029-2691-415f-aec7-435068ce2978_characterizeImage_855136_54.stderr parsl.dataflow.dflow INFO: Standard output for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/c7db2321-9f9d-446d-b129-3e22770173c6_isr_855136_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/c7db2321-9f9d-446d-b129-3e22770173c6_isr_855136_54.stderr parsl.dataflow.dflow INFO: Task 1095 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b1a216300c38dfa200a74b411392a4b with result from task 1095 parsl.dataflow.dflow DEBUG: Task 1095 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1096 has memoization hash d066c048630182efa29b5aba12770be0 parsl.dataflow.memoization INFO: Task 1096 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa453916c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1096 try 0 launched on executor multi with executor id 717 parsl.dataflow.dflow INFO: Standard output for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/06d25411-0ce6-40dc-a558-acc5bdf14a39_characterizeImage_855137_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/06d25411-0ce6-40dc-a558-acc5bdf14a39_characterizeImage_855137_37.stderr parsl.dataflow.dflow INFO: Standard output for task 1095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/f51a547a-90c2-42ef-8196-15c2575da80c_isr_855137_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/f51a547a-90c2-42ef-8196-15c2575da80c_isr_855137_37.stderr parsl.dataflow.dflow INFO: Task 1056 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a727423a2f55aa49a2aef0d076ec575f with result from task 1056 parsl.dataflow.dflow DEBUG: Task 1056 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1057 has memoization hash e8ad216e439e0fbe646e3ce7aae924ee parsl.dataflow.memoization INFO: Task 1057 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455428e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1057 try 0 launched on executor multi with executor id 718 parsl.dataflow.dflow INFO: Standard output for task 1057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/9485a19e-1a02-4359-8cc1-c41bfdaa1117_characterizeImage_855042_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/9485a19e-1a02-4359-8cc1-c41bfdaa1117_characterizeImage_855042_37.stderr parsl.dataflow.dflow INFO: Standard output for task 1056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/d6f52d2c-88a7-4799-b63f-2c52fd39028a_isr_855042_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/d6f52d2c-88a7-4799-b63f-2c52fd39028a_isr_855042_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912363 PD 16912364 PD 16912365 PD 16912366 PD 16912367 PD 16912368 PD 16912369 PD 16912370 PD 16912371 PD 16912372 PD 16912373 PD 16912374 PD 16912375 PD 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:34 proc_lsst.multi:146 [INFO] found job 16912344 in provider checkpoint proc_lsst.multi INFO: found job 16912344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 802 try 0 failed parsl.dataflow.dflow INFO: Task 802 marked for retry parsl.dataflow.dflow INFO: Standard output for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/476a5fdb-6792-4fc0-8b21-94ef66d94625_characterizeImage_855138_2.stdout parsl.dataflow.dflow INFO: Standard error for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/476a5fdb-6792-4fc0-8b21-94ef66d94625_characterizeImage_855138_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 802 has memoization hash f194a5b3aed37b05e780c4da5d373ee1 parsl.dataflow.memoization INFO: Task 802 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 802 try 1 launched on executor multi with executor id 719 parsl.dataflow.dflow INFO: Standard output for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/476a5fdb-6792-4fc0-8b21-94ef66d94625_characterizeImage_855138_2.stdout parsl.dataflow.dflow INFO: Standard error for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/476a5fdb-6792-4fc0-8b21-94ef66d94625_characterizeImage_855138_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 595 try 0 failed parsl.dataflow.dflow INFO: Task 595 marked for retry parsl.dataflow.dflow INFO: Standard output for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/69f251fe-c21e-46fd-adef-dd0cc1e49f52_characterizeImage_855043_2.stdout parsl.dataflow.dflow INFO: Standard error for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/69f251fe-c21e-46fd-adef-dd0cc1e49f52_characterizeImage_855043_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 595 has memoization hash 6a5be1ea6724fa9b116cd160e08e79b3 parsl.dataflow.memoization INFO: Task 595 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45740860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 595 try 1 launched on executor multi with executor id 720 parsl.dataflow.dflow INFO: Standard output for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/69f251fe-c21e-46fd-adef-dd0cc1e49f52_characterizeImage_855043_2.stdout parsl.dataflow.dflow INFO: Standard error for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/69f251fe-c21e-46fd-adef-dd0cc1e49f52_characterizeImage_855043_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912363 PD 16912364 PD 16912365 PD 16912366 PD 16912367 PD 16912368 PD 16912369 PD 16912370 PD 16912371 PD 16912372 PD 16912373 PD 16912374 PD 16912375 PD 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:35 proc_lsst.multi:146 [INFO] found job 16912345 in provider checkpoint proc_lsst.multi INFO: found job 16912345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375' parsl.dataflow.dflow INFO: Task 1083 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96c315dc3caf57d12d0ad4c175e6c833 with result from task 1083 parsl.dataflow.dflow DEBUG: Task 1083 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1084 has memoization hash fa1c4171a4213ec13e0cdfaf92bb1a77 parsl.dataflow.memoization INFO: Task 1084 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa453909a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1084 try 0 launched on executor multi with executor id 721 parsl.dataflow.dflow INFO: Standard output for task 1084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/767089d1-fccb-463e-ac23-0f7b17b91689_characterizeImage_855136_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/767089d1-fccb-463e-ac23-0f7b17b91689_characterizeImage_855136_19.stderr parsl.dataflow.dflow INFO: Standard output for task 1083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/373dcf03-4020-4e77-afec-8c33ce55cbe7_isr_855136_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/373dcf03-4020-4e77-afec-8c33ce55cbe7_isr_855136_19.stderr parsl.dataflow.dflow INFO: Task 1074 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6f9f8a2a4441c6a9e8da9cc152b5bd28 with result from task 1074 parsl.dataflow.dflow DEBUG: Task 1074 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1075 has memoization hash 7037946e2b120df4fc2aca0f9388d9d0 parsl.dataflow.memoization INFO: Task 1075 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa453928e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1075 try 0 launched on executor multi with executor id 722 parsl.dataflow.dflow INFO: Standard output for task 1075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/f689e8c5-cfdf-42d7-b014-22d0462efd76_characterizeImage_855136_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/f689e8c5-cfdf-42d7-b014-22d0462efd76_characterizeImage_855136_46.stderr parsl.dataflow.dflow INFO: Standard output for task 1074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/3d67f33f-f37f-49c7-a431-a83ae2dd3158_isr_855136_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/3d67f33f-f37f-49c7-a431-a83ae2dd3158_isr_855136_46.stderr parsl.dataflow.dflow INFO: Task 1113 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5498c4883e37b29d42c8342da93bddb7 with result from task 1113 parsl.dataflow.dflow DEBUG: Task 1113 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1114 has memoization hash 737803ac1b243421bbcc8f5466e7d364 parsl.dataflow.memoization INFO: Task 1114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45393600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1114 try 0 launched on executor multi with executor id 723 parsl.dataflow.dflow INFO: Standard output for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/968e1097-1bbd-49c4-98d9-71ca561a4353_characterizeImage_855137_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/968e1097-1bbd-49c4-98d9-71ca561a4353_characterizeImage_855137_56.stderr parsl.dataflow.dflow INFO: Standard output for task 1113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/5f1ec6d8-c8ab-4c57-a70d-1dfb67fafd21_isr_855137_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/5f1ec6d8-c8ab-4c57-a70d-1dfb67fafd21_isr_855137_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912363 PD 16912364 PD 16912365 PD 16912366 PD 16912367 PD 16912368 PD 16912369 PD 16912370 PD 16912371 PD 16912372 PD 16912373 PD 16912374 PD 16912375 PD 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:35 proc_lsst.multi:146 [INFO] found job 16912346 in provider checkpoint proc_lsst.multi INFO: found job 16912346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375' parsl.providers.slurm.slurm DEBUG: squeue returned 16912363 PD 16912364 PD 16912365 PD 16912366 PD 16912367 PD 16912368 PD 16912369 PD 16912370 PD 16912371 PD 16912372 PD 16912373 PD 16912374 PD 16912375 PD 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:36 proc_lsst.multi:146 [INFO] found job 16912347 in provider checkpoint proc_lsst.multi INFO: found job 16912347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375' parsl.dataflow.dflow INFO: Task 975 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9a8effc13c5988725522d3011f3d4e4 with result from task 975 parsl.dataflow.dflow DEBUG: Task 975 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 976 has memoization hash 117ca0fb41140ea06470679fc7e62a8e parsl.dataflow.memoization INFO: Task 976 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454f9260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 976 try 0 launched on executor multi with executor id 724 parsl.dataflow.dflow INFO: Standard output for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/f2d8a9c6-9b7f-4b59-872b-0ebedd4fe87b_characterizeImage_855044_59.stdout parsl.dataflow.dflow INFO: Standard error for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/f2d8a9c6-9b7f-4b59-872b-0ebedd4fe87b_characterizeImage_855044_59.stderr parsl.dataflow.dflow INFO: Standard output for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/c4ef022f-3184-4fe9-9888-ce3994969f79_isr_855044_59.stdout parsl.dataflow.dflow INFO: Standard error for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/c4ef022f-3184-4fe9-9888-ce3994969f79_isr_855044_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912363 PD 16912364 PD 16912365 PD 16912366 PD 16912367 PD 16912368 PD 16912369 PD 16912370 PD 16912371 PD 16912372 PD 16912373 PD 16912374 PD 16912375 PD 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:36 proc_lsst.multi:146 [INFO] found job 16912348 in provider checkpoint proc_lsst.multi INFO: found job 16912348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375' parsl.providers.slurm.slurm DEBUG: squeue returned 16912363 PD 16912364 PD 16912365 PD 16912366 PD 16912367 PD 16912368 PD 16912369 PD 16912370 PD 16912371 PD 16912372 PD 16912373 PD 16912374 PD 16912375 PD 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:36 proc_lsst.multi:146 [INFO] found job 16912349 in provider checkpoint proc_lsst.multi INFO: found job 16912349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375' parsl.providers.slurm.slurm DEBUG: squeue returned 16912363 PD 16912364 PD 16912365 PD 16912366 PD 16912367 PD 16912368 PD 16912369 PD 16912370 PD 16912371 PD 16912372 PD 16912373 PD 16912374 PD 16912375 PD 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:37 proc_lsst.multi:146 [INFO] found job 16912350 in provider checkpoint proc_lsst.multi INFO: found job 16912350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375' parsl.dataflow.dflow INFO: Task 1110 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 310e2334406b4255bcb1ce13fa325de7 with result from task 1110 parsl.dataflow.dflow DEBUG: Task 1110 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1111 has memoization hash 7fb4751f3ebe511ed2e632ada5200e81 parsl.dataflow.memoization INFO: Task 1111 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45392160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1111 try 0 launched on executor multi with executor id 725 parsl.dataflow.dflow INFO: Standard output for task 1111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/71ef542c-b845-4301-b3a7-05b05407e96c_characterizeImage_855138_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/71ef542c-b845-4301-b3a7-05b05407e96c_characterizeImage_855138_54.stderr parsl.dataflow.dflow INFO: Standard output for task 1110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/bef771fd-6c3a-49bb-b4a9-772db22c61e8_isr_855138_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/bef771fd-6c3a-49bb-b4a9-772db22c61e8_isr_855138_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912363 PD 16912364 PD 16912365 PD 16912366 PD 16912367 PD 16912368 PD 16912369 PD 16912370 PD 16912371 PD 16912372 PD 16912373 PD 16912374 PD 16912375 PD 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:37 proc_lsst.multi:146 [INFO] found job 16912351 in provider checkpoint proc_lsst.multi INFO: found job 16912351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375' parsl.providers.slurm.slurm DEBUG: squeue returned 16912363 PD 16912364 PD 16912365 PD 16912366 PD 16912367 PD 16912368 PD 16912369 PD 16912370 PD 16912371 PD 16912372 PD 16912373 PD 16912374 PD 16912375 PD 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:37 proc_lsst.multi:146 [INFO] found job 16912352 in provider checkpoint proc_lsst.multi INFO: found job 16912352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375' parsl.providers.slurm.slurm DEBUG: squeue returned 16912363 PD 16912364 PD 16912365 PD 16912366 PD 16912367 PD 16912368 PD 16912369 PD 16912370 PD 16912371 PD 16912372 PD 16912373 PD 16912374 PD 16912375 PD 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:38 proc_lsst.multi:146 [INFO] found job 16912353 in provider checkpoint proc_lsst.multi INFO: found job 16912353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375' parsl.dataflow.dflow INFO: Task 1101 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 52753e7e8068cd9c1805481b6f29659b with result from task 1101 parsl.dataflow.dflow DEBUG: Task 1101 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1102 has memoization hash a8fba6ac30cfd3dc99f53fce6ccd2ccf parsl.dataflow.memoization INFO: Task 1102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45391d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1102 try 0 launched on executor multi with executor id 726 parsl.dataflow.dflow INFO: Standard output for task 1102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/95e4f8e6-3262-44dd-9945-ca0163e2da91_characterizeImage_855136_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/95e4f8e6-3262-44dd-9945-ca0163e2da91_characterizeImage_855136_18.stderr parsl.dataflow.dflow INFO: Standard output for task 1101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/844c6b77-1477-41e0-a3ef-f0a71cf780b3_isr_855136_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/844c6b77-1477-41e0-a3ef-f0a71cf780b3_isr_855136_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912363 PD 16912364 PD 16912365 PD 16912366 PD 16912367 PD 16912368 PD 16912369 PD 16912370 PD 16912371 PD 16912372 PD 16912373 PD 16912374 PD 16912375 PD 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:39 proc_lsst.multi:146 [INFO] found job 16912354 in provider checkpoint proc_lsst.multi INFO: found job 16912354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 118 try 0 failed parsl.dataflow.dflow INFO: Task 118 marked for retry parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/f4ba88e8-9ec1-4d37-a2e0-cdd18c3b7056_characterizeImage_855042_2.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/f4ba88e8-9ec1-4d37-a2e0-cdd18c3b7056_characterizeImage_855042_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 118 has memoization hash 90171a04074af488fac4cfc595aa2314 parsl.dataflow.memoization INFO: Task 118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f5f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 118 try 1 launched on executor multi with executor id 727 parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/f4ba88e8-9ec1-4d37-a2e0-cdd18c3b7056_characterizeImage_855042_2.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/f4ba88e8-9ec1-4d37-a2e0-cdd18c3b7056_characterizeImage_855042_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912363 PD 16912364 PD 16912365 PD 16912366 PD 16912367 PD 16912368 PD 16912369 PD 16912370 PD 16912371 PD 16912372 PD 16912373 PD 16912374 PD 16912375 PD 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:39 proc_lsst.multi:146 [INFO] found job 16912355 in provider checkpoint proc_lsst.multi INFO: found job 16912355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375' parsl.providers.slurm.slurm DEBUG: squeue returned 16912363 PD 16912364 PD 16912365 PD 16912366 PD 16912367 PD 16912368 PD 16912369 PD 16912370 PD 16912371 PD 16912372 PD 16912373 PD 16912374 PD 16912375 PD 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:40 proc_lsst.multi:146 [INFO] found job 16912356 in provider checkpoint proc_lsst.multi INFO: found job 16912356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375' parsl.providers.slurm.slurm DEBUG: squeue returned 16912363 PD 16912364 PD 16912365 PD 16912366 PD 16912367 PD 16912368 PD 16912369 PD 16912370 PD 16912371 PD 16912372 PD 16912373 PD 16912374 PD 16912375 PD 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:40 proc_lsst.multi:146 [INFO] found job 16912359 in provider checkpoint proc_lsst.multi INFO: found job 16912359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375' parsl.providers.slurm.slurm DEBUG: squeue returned 16912363 PD 16912364 PD 16912365 PD 16912366 PD 16912367 PD 16912368 PD 16912369 PD 16912370 PD 16912371 PD 16912372 PD 16912373 PD 16912374 PD 16912375 PD 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:41 proc_lsst.multi:146 [INFO] found job 16912360 in provider checkpoint proc_lsst.multi INFO: found job 16912360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375' parsl.dataflow.dflow INFO: Task 417 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a963f419d03d5d648445378946a4c6d with result from task 417 parsl.dataflow.dflow DEBUG: Task 417 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 418 has memoization hash 1ce77ddc1e7ee59544c53d10046ab314 parsl.dataflow.memoization INFO: Task 418 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a47c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 418 try 0 launched on executor multi with executor id 728 parsl.dataflow.dflow INFO: Standard output for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/e4777fdb-64ac-4696-8b33-e492962b1623_characterizeImage_855137_59.stdout parsl.dataflow.dflow INFO: Standard error for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/e4777fdb-64ac-4696-8b33-e492962b1623_characterizeImage_855137_59.stderr parsl.dataflow.dflow INFO: Standard output for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/de8a68b1-ce98-4db0-8e04-c48db6703d53_isr_855137_59.stdout parsl.dataflow.dflow INFO: Standard error for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/de8a68b1-ce98-4db0-8e04-c48db6703d53_isr_855137_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912363 PD 16912364 PD 16912365 PD 16912366 PD 16912367 PD 16912368 PD 16912369 PD 16912370 PD 16912371 PD 16912372 PD 16912373 PD 16912374 PD 16912375 PD 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:42 proc_lsst.multi:146 [INFO] found job 16912361 in provider checkpoint proc_lsst.multi INFO: found job 16912361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375' parsl.providers.slurm.slurm DEBUG: squeue returned 16912363 PD 16912364 PD 16912365 PD 16912366 PD 16912367 PD 16912368 PD 16912369 PD 16912370 PD 16912371 PD 16912372 PD 16912373 PD 16912374 PD 16912375 PD 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:42 proc_lsst.multi:146 [INFO] found job 16912362 in provider checkpoint proc_lsst.multi INFO: found job 16912362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375' parsl.providers.slurm.slurm DEBUG: squeue returned 16912363 PD 16912364 PD 16912365 PD 16912366 PD 16912367 PD 16912368 PD 16912369 PD 16912370 PD 16912371 PD 16912372 PD 16912373 PD 16912374 PD 16912375 PD 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:43 proc_lsst.multi:146 [INFO] found job 16912363 in provider checkpoint proc_lsst.multi INFO: found job 16912363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375' parsl.providers.slurm.slurm DEBUG: squeue returned 16912363 PD 16912364 PD 16912365 PD 16912366 PD 16912367 PD 16912368 PD 16912369 PD 16912370 PD 16912371 PD 16912372 PD 16912373 PD 16912374 PD 16912375 PD 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:43 proc_lsst.multi:146 [INFO] found job 16912364 in provider checkpoint proc_lsst.multi INFO: found job 16912364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375' parsl.providers.slurm.slurm DEBUG: squeue returned 16912363 PD 16912364 PD 16912365 PD 16912366 PD 16912367 PD 16912368 PD 16912369 PD 16912370 PD 16912371 PD 16912372 PD 16912373 PD 16912374 PD 16912375 PD 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:44 proc_lsst.multi:146 [INFO] found job 16912365 in provider checkpoint proc_lsst.multi INFO: found job 16912365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 874 try 0 failed parsl.dataflow.dflow INFO: Task 874 marked for retry parsl.dataflow.dflow INFO: Standard output for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/ec5235e9-6c9e-47d1-babb-4ec2b9224d0f_characterizeImage_855136_2.stdout parsl.dataflow.dflow INFO: Standard error for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/ec5235e9-6c9e-47d1-babb-4ec2b9224d0f_characterizeImage_855136_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 874 has memoization hash d5ee3c2628c49d0e826ae7f7da60ef7f parsl.dataflow.memoization INFO: Task 874 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45667100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 874 try 1 launched on executor multi with executor id 729 parsl.dataflow.dflow INFO: Standard output for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/ec5235e9-6c9e-47d1-babb-4ec2b9224d0f_characterizeImage_855136_2.stdout parsl.dataflow.dflow INFO: Standard error for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/ec5235e9-6c9e-47d1-babb-4ec2b9224d0f_characterizeImage_855136_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 919 try 0 failed parsl.dataflow.dflow INFO: Task 919 marked for retry parsl.dataflow.dflow INFO: Standard output for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/80704541-bab4-4b60-9c40-45116b4567c1_characterizeImage_855044_2.stdout parsl.dataflow.dflow INFO: Standard error for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/80704541-bab4-4b60-9c40-45116b4567c1_characterizeImage_855044_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 919 has memoization hash 11b95bba4e0daed4db9efaf1de5e401c parsl.dataflow.memoization INFO: Task 919 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4549a8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 919 try 1 launched on executor multi with executor id 730 parsl.dataflow.dflow INFO: Standard output for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/80704541-bab4-4b60-9c40-45116b4567c1_characterizeImage_855044_2.stdout parsl.dataflow.dflow INFO: Standard error for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/80704541-bab4-4b60-9c40-45116b4567c1_characterizeImage_855044_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 391 try 0 failed parsl.dataflow.dflow INFO: Task 391 marked for retry parsl.dataflow.dflow INFO: Standard output for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/d7f79a9c-f7aa-42a3-84c1-ce23407ffc8d_characterizeImage_855137_2.stdout parsl.dataflow.dflow INFO: Standard error for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/d7f79a9c-f7aa-42a3-84c1-ce23407ffc8d_characterizeImage_855137_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 391 has memoization hash a97b8755c078a2e49ccc0564639c1bb4 parsl.dataflow.memoization INFO: Task 391 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4687bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 391 try 1 launched on executor multi with executor id 731 parsl.dataflow.dflow INFO: Standard output for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/d7f79a9c-f7aa-42a3-84c1-ce23407ffc8d_characterizeImage_855137_2.stdout parsl.dataflow.dflow INFO: Standard error for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/d7f79a9c-f7aa-42a3-84c1-ce23407ffc8d_characterizeImage_855137_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912363 PD 16912364 PD 16912365 PD 16912366 PD 16912367 PD 16912368 PD 16912369 PD 16912370 PD 16912371 PD 16912372 PD 16912373 PD 16912374 PD 16912375 PD 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:44 proc_lsst.multi:146 [INFO] found job 16912366 in provider checkpoint proc_lsst.multi INFO: found job 16912366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375' parsl.providers.slurm.slurm DEBUG: squeue returned 16912363 PD 16912364 PD 16912365 PD 16912366 PD 16912367 PD 16912368 PD 16912369 PD 16912370 PD 16912371 PD 16912372 PD 16912373 PD 16912374 PD 16912375 PD 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:45 proc_lsst.multi:146 [INFO] found job 16912367 in provider checkpoint proc_lsst.multi INFO: found job 16912367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375' parsl.providers.slurm.slurm DEBUG: squeue returned 16912363 PD 16912364 PD 16912365 PD 16912366 PD 16912367 PD 16912368 PD 16912369 PD 16912370 PD 16912371 PD 16912372 PD 16912373 PD 16912374 PD 16912375 PD 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:46 proc_lsst.multi:146 [INFO] found job 16912368 in provider checkpoint proc_lsst.multi INFO: found job 16912368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375' parsl.providers.slurm.slurm DEBUG: squeue returned 16912363 PD 16912364 PD 16912365 PD 16912366 PD 16912367 PD 16912368 PD 16912369 PD 16912370 PD 16912371 PD 16912372 PD 16912373 PD 16912374 PD 16912375 PD 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:47 proc_lsst.multi:146 [INFO] found job 16912369 in provider checkpoint proc_lsst.multi INFO: found job 16912369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375' parsl.providers.slurm.slurm DEBUG: squeue returned 16912363 PD 16912364 PD 16912365 PD 16912366 PD 16912367 PD 16912368 PD 16912369 PD 16912370 PD 16912371 PD 16912372 PD 16912373 PD 16912374 PD 16912375 PD 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:48 proc_lsst.multi:146 [INFO] found job 16912370 in provider checkpoint proc_lsst.multi INFO: found job 16912370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375' parsl.providers.slurm.slurm DEBUG: squeue returned 16912363 PD 16912364 PD 16912365 PD 16912366 PD 16912367 PD 16912368 PD 16912369 PD 16912370 PD 16912371 PD 16912372 PD 16912373 PD 16912374 PD 16912375 PD 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:48 proc_lsst.multi:146 [INFO] found job 16912371 in provider checkpoint proc_lsst.multi INFO: found job 16912371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375' parsl.dataflow.dflow INFO: Task 423 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ef5506ed50f749b0bae771bcedce166 with result from task 423 parsl.dataflow.dflow DEBUG: Task 423 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 424 has memoization hash 10e074e2aa0cf1704bc40417b732483a parsl.dataflow.memoization INFO: Task 424 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a56c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 424 try 0 launched on executor multi with executor id 732 parsl.dataflow.dflow INFO: Standard output for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/d263fdd4-f7dc-414d-b52a-da9989834544_characterizeImage_855044_51.stdout parsl.dataflow.dflow INFO: Standard error for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/d263fdd4-f7dc-414d-b52a-da9989834544_characterizeImage_855044_51.stderr parsl.dataflow.dflow INFO: Standard output for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/e4bbdce0-0896-41b1-9aaf-85ebd3a09d52_isr_855044_51.stdout parsl.dataflow.dflow INFO: Standard error for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/e4bbdce0-0896-41b1-9aaf-85ebd3a09d52_isr_855044_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912363 PD 16912364 PD 16912365 PD 16912366 PD 16912367 PD 16912368 PD 16912369 PD 16912370 PD 16912371 PD 16912372 PD 16912373 PD 16912374 PD 16912375 PD 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:49 proc_lsst.multi:146 [INFO] found job 16912372 in provider checkpoint proc_lsst.multi INFO: found job 16912372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375' parsl.dataflow.dflow INFO: Task 762 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7f03924dc4a2bec888aa6902d162bb2 with result from task 762 parsl.dataflow.dflow DEBUG: Task 762 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 763 has memoization hash eba77e20e37a32b66a51fa94fe977dea parsl.dataflow.memoization INFO: Task 763 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d3600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 763 try 0 launched on executor multi with executor id 733 parsl.dataflow.dflow INFO: Standard output for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/ed3f38e4-304a-46f6-b4b8-93713ae9c2e7_characterizeImage_855137_18.stdout parsl.dataflow.dflow INFO: Standard error for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/ed3f38e4-304a-46f6-b4b8-93713ae9c2e7_characterizeImage_855137_18.stderr parsl.dataflow.dflow INFO: Standard output for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/6fe451c3-0f6c-482d-8c1c-0d44e9fe0bbe_isr_855137_18.stdout parsl.dataflow.dflow INFO: Standard error for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/6fe451c3-0f6c-482d-8c1c-0d44e9fe0bbe_isr_855137_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912363 PD 16912364 PD 16912365 PD 16912366 PD 16912367 PD 16912368 PD 16912369 PD 16912370 PD 16912371 PD 16912372 PD 16912373 PD 16912374 PD 16912375 PD 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:50 proc_lsst.multi:146 [INFO] found job 16912373 in provider checkpoint proc_lsst.multi INFO: found job 16912373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375' parsl.providers.slurm.slurm DEBUG: squeue returned 16912363 PD 16912364 PD 16912365 PD 16912366 PD 16912367 PD 16912368 PD 16912369 PD 16912370 PD 16912371 PD 16912372 PD 16912373 PD 16912374 PD 16912375 PD 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:51 proc_lsst.multi:146 [INFO] found job 16912374 in provider checkpoint proc_lsst.multi INFO: found job 16912374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375' parsl.dataflow.dflow INFO: Task 969 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed053a8fd9124ccf2f6780eb505bd3ce with result from task 969 parsl.dataflow.dflow DEBUG: Task 969 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 970 has memoization hash ca29a887279439453680fdc7b649d988 parsl.dataflow.memoization INFO: Task 970 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454f8ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 970 try 0 launched on executor multi with executor id 734 parsl.dataflow.dflow INFO: Standard output for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/c409a54a-f82f-433f-b837-c9fcb5c96001_characterizeImage_855042_61.stdout parsl.dataflow.dflow INFO: Standard error for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/c409a54a-f82f-433f-b837-c9fcb5c96001_characterizeImage_855042_61.stderr parsl.dataflow.dflow INFO: Standard output for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/2fb5a2b3-9e28-4be0-84dd-ca05ec516fd1_isr_855042_61.stdout parsl.dataflow.dflow INFO: Standard error for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855042/2fb5a2b3-9e28-4be0-84dd-ca05ec516fd1_isr_855042_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912375 PD 16912374 PD 16912373 PD 16912372 PD 16912371 PD 16912370 PD 16912369 PD 16912368 PD 16912367 PD 16912366 PD 16912365 PD 16912364 PD 16912363 PD 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:53 proc_lsst.multi:146 [INFO] found job 16912375 in provider checkpoint proc_lsst.multi INFO: found job 16912375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375' parsl.providers.slurm.slurm DEBUG: squeue returned 16912375 PD 16912374 PD 16912373 PD 16912372 PD 16912371 PD 16912370 PD 16912369 PD 16912368 PD 16912367 PD 16912366 PD 16912365 PD 16912364 PD 16912363 PD 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 92, active_tasks = 372 parsl.dataflow.strategy DEBUG: Executor multi has 372 active tasks, 79/13 running/pending blocks, and 241 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 92 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 1 more blocks parsl.executors.status_handling INFO: Scaling out by 1 blocks parsl.executors.status_handling INFO: Allocated block ID 93 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-93 2024-03-08 07:16:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-93 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-93 2024-03-08 07:16:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 07:16:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 07:16:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 07:16:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:16:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 07:16:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 07:16:56 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:16:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 07:16:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 07:16:56 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:16:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 07:16:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 07:16:56 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 07:16:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 36 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 36 2024-03-08 07:16:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:16:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3064,10.64.65.64,127.0.0.1,10.64.129.64,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54598 --result_port=54018 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 07:16:58 proc_lsst.multi:170 [INFO] [multi] job_id 16912378 proc_lsst.multi INFO: [multi] job_id 16912378 2024-03-08 07:16:58 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 37 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 37 2024-03-08 07:16:58 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 07:16:58 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16912378 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16912378 parsl.executors.status_handling DEBUG: Launched block 93 on executor multi with job ID 16912378 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 2024-03-08 07:16:58 proc_lsst.multi:146 [INFO] found job 75775 in provider local proc_lsst.multi INFO: found job 75775 in provider local 2024-03-08 07:16:58 proc_lsst.multi:146 [INFO] found job 16912283 in provider astro proc_lsst.multi INFO: found job 16912283 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:16:59 proc_lsst.multi:146 [INFO] found job 16912284 in provider astro proc_lsst.multi INFO: found job 16912284 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:00 proc_lsst.multi:146 [INFO] found job 16912285 in provider astro proc_lsst.multi INFO: found job 16912285 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:01 proc_lsst.multi:146 [INFO] found job 16912286 in provider astro proc_lsst.multi INFO: found job 16912286 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:02 proc_lsst.multi:146 [INFO] found job 16912287 in provider astro proc_lsst.multi INFO: found job 16912287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:03 proc_lsst.multi:146 [INFO] found job 16912288 in provider astro proc_lsst.multi INFO: found job 16912288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:03 proc_lsst.multi:146 [INFO] found job 16912289 in provider astro proc_lsst.multi INFO: found job 16912289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:04 proc_lsst.multi:146 [INFO] found job 16912290 in provider astro proc_lsst.multi INFO: found job 16912290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:05 proc_lsst.multi:146 [INFO] found job 16912291 in provider astro proc_lsst.multi INFO: found job 16912291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:06 proc_lsst.multi:146 [INFO] found job 16912292 in provider astro proc_lsst.multi INFO: found job 16912292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:07 proc_lsst.multi:146 [INFO] found job 16912293 in provider astro proc_lsst.multi INFO: found job 16912293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 681 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eaf63182117ade877da40e9f81c96403 with result from task 681 parsl.dataflow.dflow DEBUG: Task 681 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 682 has memoization hash c02d8945968feac74ddadbe302a0e756 parsl.dataflow.memoization INFO: Task 682 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 682 try 0 launched on executor multi with executor id 735 parsl.dataflow.dflow INFO: Standard output for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/eb792d5b-e5a4-43ca-8d0d-3d614ff10633_characterizeImage_855043_28.stdout parsl.dataflow.dflow INFO: Standard error for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/eb792d5b-e5a4-43ca-8d0d-3d614ff10633_characterizeImage_855043_28.stderr parsl.dataflow.dflow INFO: Standard output for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/61f56ff7-8046-4852-bcff-29e80fdae1f3_isr_855043_28.stdout parsl.dataflow.dflow INFO: Standard error for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/61f56ff7-8046-4852-bcff-29e80fdae1f3_isr_855043_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:08 proc_lsst.multi:146 [INFO] found job 16912294 in provider astro proc_lsst.multi INFO: found job 16912294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 666 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53e97261018009780ef7d126ed1cb0f5 with result from task 666 parsl.dataflow.dflow DEBUG: Task 666 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 667 has memoization hash 98ea1bca8cd3e25f982427233db3d17d parsl.dataflow.memoization INFO: Task 667 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 667 try 0 launched on executor multi with executor id 736 parsl.dataflow.dflow INFO: Standard output for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/e1119258-8d35-4ea0-8f58-2f4924344592_characterizeImage_855043_18.stdout parsl.dataflow.dflow INFO: Standard error for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/e1119258-8d35-4ea0-8f58-2f4924344592_characterizeImage_855043_18.stderr parsl.dataflow.dflow INFO: Standard output for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/af10f13f-22b7-49a7-bd72-385a8d7ec726_isr_855043_18.stdout parsl.dataflow.dflow INFO: Standard error for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/af10f13f-22b7-49a7-bd72-385a8d7ec726_isr_855043_18.stderr parsl.dataflow.dflow INFO: Task 684 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 463891bf1da200cadda72579d0755442 with result from task 684 parsl.dataflow.dflow DEBUG: Task 684 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 685 has memoization hash 4250d8da97c588b38a6b80dc925ba1b1 parsl.dataflow.memoization INFO: Task 685 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 685 try 0 launched on executor multi with executor id 737 parsl.dataflow.dflow INFO: Standard output for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/0f0888ee-101b-41ea-b843-fe9d2e5169a9_characterizeImage_855044_52.stdout parsl.dataflow.dflow INFO: Standard error for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/0f0888ee-101b-41ea-b843-fe9d2e5169a9_characterizeImage_855044_52.stderr parsl.dataflow.dflow INFO: Standard output for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/379d8e0b-f22d-4e8b-9780-920d4b25183d_isr_855044_52.stdout parsl.dataflow.dflow INFO: Standard error for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/379d8e0b-f22d-4e8b-9780-920d4b25183d_isr_855044_52.stderr parsl.dataflow.dflow INFO: Task 636 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d06f8717c2449067698c928fc99a31b7 with result from task 636 parsl.dataflow.dflow DEBUG: Task 636 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 637 has memoization hash f30f6235571db374d6a0fff9a103cbcb parsl.dataflow.memoization INFO: Task 637 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45743f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 637 try 0 launched on executor multi with executor id 738 parsl.dataflow.dflow INFO: Standard output for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/9b66bc03-4706-4333-b3f0-b409d71e2acd_characterizeImage_855044_50.stdout parsl.dataflow.dflow INFO: Standard error for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/9b66bc03-4706-4333-b3f0-b409d71e2acd_characterizeImage_855044_50.stderr parsl.dataflow.dflow INFO: Standard output for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/be5876af-fddf-4e4f-8c5a-0340b13351b3_isr_855044_50.stdout parsl.dataflow.dflow INFO: Standard error for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/be5876af-fddf-4e4f-8c5a-0340b13351b3_isr_855044_50.stderr parsl.dataflow.dflow INFO: Task 630 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae3c5bf585a12bf380cf45cde8939a99 with result from task 630 parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:08 proc_lsst.multi:146 [INFO] found job 16912295 in provider astro proc_lsst.multi INFO: found job 16912295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow DEBUG: Task 630 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 631 has memoization hash e784c2eca78637ac23981f726aff79b4 parsl.dataflow.memoization INFO: Task 631 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45743ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 631 try 0 launched on executor multi with executor id 739 parsl.dataflow.dflow INFO: Standard output for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/8af16452-4b58-4109-894e-0f033e6faafc_characterizeImage_855044_30.stdout parsl.dataflow.dflow INFO: Standard error for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/8af16452-4b58-4109-894e-0f033e6faafc_characterizeImage_855044_30.stderr parsl.dataflow.dflow INFO: Standard output for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/7a047668-758e-468f-9421-637decc85fc1_isr_855044_30.stdout parsl.dataflow.dflow INFO: Standard error for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/7a047668-758e-468f-9421-637decc85fc1_isr_855044_30.stderr parsl.dataflow.dflow INFO: Task 618 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eeeac2118c829491302d72bbc3126ade with result from task 618 parsl.dataflow.dflow DEBUG: Task 618 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 619 has memoization hash 58fd415b8fe3b1e8d74ac2565e4a6e2c parsl.dataflow.memoization INFO: Task 619 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45741f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 619 try 0 launched on executor multi with executor id 740 parsl.dataflow.dflow INFO: Standard output for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/fb0e0002-eb8e-4b0b-b6f9-3314ddc17a01_characterizeImage_855136_41.stdout parsl.dataflow.dflow INFO: Standard error for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/fb0e0002-eb8e-4b0b-b6f9-3314ddc17a01_characterizeImage_855136_41.stderr parsl.dataflow.dflow INFO: Standard output for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/3c45c250-5c01-481f-9886-78c1cb283b48_isr_855136_41.stdout parsl.dataflow.dflow INFO: Standard error for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/3c45c250-5c01-481f-9886-78c1cb283b48_isr_855136_41.stderr parsl.dataflow.dflow INFO: Task 624 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b98f5a9400ed73dc0bb225e20943668a with result from task 624 parsl.dataflow.dflow DEBUG: Task 624 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 625 has memoization hash e4ca2012112cba1b678751ee713b9c86 parsl.dataflow.memoization INFO: Task 625 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45742520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 625 try 0 launched on executor multi with executor id 741 parsl.dataflow.dflow INFO: Standard output for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/2ef4106b-603f-403e-a01d-639ad27e9fcb_characterizeImage_855138_34.stdout parsl.dataflow.dflow INFO: Standard error for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/2ef4106b-603f-403e-a01d-639ad27e9fcb_characterizeImage_855138_34.stderr parsl.dataflow.dflow INFO: Standard output for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/f78f9864-d7dd-40ae-a0d2-4c1596c85ea7_isr_855138_34.stdout parsl.dataflow.dflow INFO: Standard error for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/f78f9864-d7dd-40ae-a0d2-4c1596c85ea7_isr_855138_34.stderr parsl.dataflow.dflow INFO: Task 678 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6faffc285b5819da6b033bed64e754ed with result from task 678 parsl.dataflow.dflow DEBUG: Task 678 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 679 has memoization hash 754a6fb44dc721493ebbaa80ee52fe05 parsl.dataflow.memoization INFO: Task 679 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 679 try 0 launched on executor multi with executor id 742 parsl.dataflow.dflow INFO: Standard output for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/4c80327b-7030-4b69-975e-7e89ddfe4444_characterizeImage_855136_11.stdout parsl.dataflow.dflow INFO: Standard error for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/4c80327b-7030-4b69-975e-7e89ddfe4444_characterizeImage_855136_11.stderr parsl.dataflow.dflow INFO: Standard output for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/ea4df522-cb2f-4235-83a4-c1d408b1d2bb_isr_855136_11.stdout parsl.dataflow.dflow INFO: Standard error for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/ea4df522-cb2f-4235-83a4-c1d408b1d2bb_isr_855136_11.stderr parsl.dataflow.dflow INFO: Task 621 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 875d93fa9ccc8ce0f9c61ebef8878647 with result from task 621 parsl.dataflow.dflow DEBUG: Task 621 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 622 has memoization hash 6ddf7f7138d9aaf0e49096a5cfe8f4ca parsl.dataflow.memoization INFO: Task 622 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45742840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 622 try 0 launched on executor multi with executor id 743 parsl.dataflow.dflow INFO: Standard output for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/b97e250c-4e6a-4f37-86d7-a4a7c4f9ac6c_characterizeImage_855136_12.stdout parsl.dataflow.dflow INFO: Standard error for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/b97e250c-4e6a-4f37-86d7-a4a7c4f9ac6c_characterizeImage_855136_12.stderr parsl.dataflow.dflow INFO: Standard output for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/5ac2c48d-5ab3-41d0-903d-da9dc12c3dbd_isr_855136_12.stdout parsl.dataflow.dflow INFO: Standard error for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/5ac2c48d-5ab3-41d0-903d-da9dc12c3dbd_isr_855136_12.stderr parsl.dataflow.dflow INFO: Task 687 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry abe1781027a7e6ee598cc46b5fbafb24 with result from task 687 parsl.dataflow.dflow DEBUG: Task 687 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 688 has memoization hash c6b45dcf2621d197b005ce99ab306736 parsl.dataflow.memoization INFO: Task 688 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 688 try 0 launched on executor multi with executor id 744 parsl.dataflow.dflow INFO: Standard output for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/bb4cb75c-1e13-42f5-bf68-6986a2c18fd4_characterizeImage_855136_15.stdout parsl.dataflow.dflow INFO: Standard error for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/bb4cb75c-1e13-42f5-bf68-6986a2c18fd4_characterizeImage_855136_15.stderr parsl.dataflow.dflow INFO: Standard output for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/cc103e0d-f02b-4611-b54c-30f2558e9bea_isr_855136_15.stdout parsl.dataflow.dflow INFO: Standard error for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855136/cc103e0d-f02b-4611-b54c-30f2558e9bea_isr_855136_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:09 proc_lsst.multi:146 [INFO] found job 16912296 in provider astro proc_lsst.multi INFO: found job 16912296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 672 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1282e725cc7d1feb657f3339c7a5012d with result from task 672 parsl.dataflow.dflow DEBUG: Task 672 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 673 has memoization hash dca7fab4b28fa9b9ce6c00ebbab33c97 parsl.dataflow.memoization INFO: Task 673 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 673 try 0 launched on executor multi with executor id 745 parsl.dataflow.dflow INFO: Standard output for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/2c676aad-45b1-47a2-833e-2e0eb389ad63_characterizeImage_855043_16.stdout parsl.dataflow.dflow INFO: Standard error for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/2c676aad-45b1-47a2-833e-2e0eb389ad63_characterizeImage_855043_16.stderr parsl.dataflow.dflow INFO: Standard output for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/fb78e50d-9cfa-455d-9803-184aa0fb616b_isr_855043_16.stdout parsl.dataflow.dflow INFO: Standard error for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855043/fb78e50d-9cfa-455d-9803-184aa0fb616b_isr_855043_16.stderr parsl.dataflow.dflow INFO: Task 675 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 429f994fa841117fa57d02bfd851759e with result from task 675 parsl.dataflow.dflow DEBUG: Task 675 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 676 has memoization hash 3647bf93b5bf7217d7b7242b2d471a26 parsl.dataflow.memoization INFO: Task 676 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 676 try 0 launched on executor multi with executor id 746 parsl.dataflow.dflow INFO: Standard output for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/f4941610-c57a-4966-a9bd-e74d8c74465e_characterizeImage_855044_43.stdout parsl.dataflow.dflow INFO: Standard error for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/f4941610-c57a-4966-a9bd-e74d8c74465e_characterizeImage_855044_43.stderr parsl.dataflow.dflow INFO: Standard output for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/3645ad0c-3140-4e83-a056-9a0ab7e0f86e_isr_855044_43.stdout parsl.dataflow.dflow INFO: Standard error for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/3645ad0c-3140-4e83-a056-9a0ab7e0f86e_isr_855044_43.stderr parsl.dataflow.dflow INFO: Task 627 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 31c55e94be899179bdab687fcfe605d2 with result from task 627 parsl.dataflow.dflow DEBUG: Task 627 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 628 has memoization hash bc74d3d87b3979c6ba1e602205246fa9 parsl.dataflow.memoization INFO: Task 628 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45743880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 628 try 0 launched on executor multi with executor id 747 parsl.dataflow.dflow INFO: Standard output for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/48f64f52-20be-4871-a120-c66b8049ad9b_characterizeImage_855137_60.stdout parsl.dataflow.dflow INFO: Standard error for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/48f64f52-20be-4871-a120-c66b8049ad9b_characterizeImage_855137_60.stderr parsl.dataflow.dflow INFO: Standard output for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/ab2b3952-9747-4552-9306-34b00612399c_isr_855137_60.stdout parsl.dataflow.dflow INFO: Standard error for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/ab2b3952-9747-4552-9306-34b00612399c_isr_855137_60.stderr parsl.dataflow.dflow INFO: Task 639 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f91c3ed579234fe73293194a8d0bb040 with result from task 639 parsl.dataflow.dflow DEBUG: Task 639 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 640 has memoization hash 5a47e49bf392be9628d57e27b602310a parsl.dataflow.memoization INFO: Task 640 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45740360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 640 try 0 launched on executor multi with executor id 748 parsl.dataflow.dflow INFO: Standard output for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/fa3f3328-e876-474f-9636-78c578e73243_characterizeImage_855138_47.stdout parsl.dataflow.dflow INFO: Standard error for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/fa3f3328-e876-474f-9636-78c578e73243_characterizeImage_855138_47.stderr parsl.dataflow.dflow INFO: Standard output for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/6b7905cd-744c-483b-a446-6c4d81b9ba6a_isr_855138_47.stdout parsl.dataflow.dflow INFO: Standard error for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855138/6b7905cd-744c-483b-a446-6c4d81b9ba6a_isr_855138_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:10 proc_lsst.multi:146 [INFO] found job 16912297 in provider astro proc_lsst.multi INFO: found job 16912297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:11 proc_lsst.multi:146 [INFO] found job 16912298 in provider astro proc_lsst.multi INFO: found job 16912298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 669 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 880a4428a49ef3ce80021ccf44730c1a with result from task 669 parsl.dataflow.dflow DEBUG: Task 669 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 670 has memoization hash 1ec92eb4f257598a16e6a452a4d5456c parsl.dataflow.memoization INFO: Task 670 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 670 try 0 launched on executor multi with executor id 749 parsl.dataflow.dflow INFO: Standard output for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/e8a1316a-9416-4158-bd33-ea6782e35678_characterizeImage_855137_55.stdout parsl.dataflow.dflow INFO: Standard error for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/e8a1316a-9416-4158-bd33-ea6782e35678_characterizeImage_855137_55.stderr parsl.dataflow.dflow INFO: Standard output for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/1a89a24f-2e73-4a5e-ad74-28d4e38ea198_isr_855137_55.stdout parsl.dataflow.dflow INFO: Standard error for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855137/1a89a24f-2e73-4a5e-ad74-28d4e38ea198_isr_855137_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:12 proc_lsst.multi:146 [INFO] found job 16912299 in provider astro proc_lsst.multi INFO: found job 16912299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 633 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 40a253af77c35b9e13dd6815fa28b211 with result from task 633 parsl.dataflow.dflow DEBUG: Task 633 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 634 has memoization hash f3646231171e0ca43a6f5ab7ca1c0b42 parsl.dataflow.memoization INFO: Task 634 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45743ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 634 try 0 launched on executor multi with executor id 750 parsl.dataflow.dflow INFO: Standard output for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/193fd599-2bd6-42f8-8431-c9e95745e8a1_characterizeImage_855044_37.stdout parsl.dataflow.dflow INFO: Standard error for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/193fd599-2bd6-42f8-8431-c9e95745e8a1_characterizeImage_855044_37.stderr parsl.dataflow.dflow INFO: Standard output for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/283c6682-faee-4bc8-8224-15ac6c2e3e10_isr_855044_37.stdout parsl.dataflow.dflow INFO: Standard error for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/isr/855044/283c6682-faee-4bc8-8224-15ac6c2e3e10_isr_855044_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:13 proc_lsst.multi:146 [INFO] found job 16912300 in provider astro proc_lsst.multi INFO: found job 16912300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:13 proc_lsst.multi:146 [INFO] found job 16912301 in provider astro proc_lsst.multi INFO: found job 16912301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:14 proc_lsst.multi:146 [INFO] found job 16912302 in provider astro proc_lsst.multi INFO: found job 16912302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:15 proc_lsst.multi:146 [INFO] found job 16912303 in provider astro proc_lsst.multi INFO: found job 16912303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:16 proc_lsst.multi:146 [INFO] found job 16912304 in provider astro proc_lsst.multi INFO: found job 16912304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:17 proc_lsst.multi:146 [INFO] found job 16912305 in provider astro proc_lsst.multi INFO: found job 16912305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:18 proc_lsst.multi:146 [INFO] found job 16912306 in provider astro proc_lsst.multi INFO: found job 16912306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:18 proc_lsst.multi:146 [INFO] found job 16912307 in provider astro proc_lsst.multi INFO: found job 16912307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:19 proc_lsst.multi:146 [INFO] found job 16912308 in provider astro proc_lsst.multi INFO: found job 16912308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:20 proc_lsst.multi:146 [INFO] found job 16912309 in provider astro proc_lsst.multi INFO: found job 16912309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:21 proc_lsst.multi:146 [INFO] found job 16912310 in provider astro proc_lsst.multi INFO: found job 16912310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:22 proc_lsst.multi:146 [INFO] found job 16912311 in provider astro proc_lsst.multi INFO: found job 16912311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 814 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 550cd89d33aa0aaa3845dcbb329e74e0 with result from task 814 parsl.dataflow.dflow DEBUG: Task 814 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 815 has memoization hash f68922ca7ecf948b5cb4f5bd01b5dca1 parsl.dataflow.memoization INFO: Task 815 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 815 try 0 launched on executor multi with executor id 751 parsl.dataflow.dflow INFO: Standard output for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/f6aa7f37-4add-4cd8-8d1d-eeac59446ae9_calibrate_855043_31.stdout parsl.dataflow.dflow INFO: Standard error for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/f6aa7f37-4add-4cd8-8d1d-eeac59446ae9_calibrate_855043_31.stderr parsl.dataflow.dflow INFO: Standard output for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/2d02d6b8-8e86-44dc-845b-25351ab8f58e_characterizeImage_855043_31.stdout parsl.dataflow.dflow INFO: Standard error for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/2d02d6b8-8e86-44dc-845b-25351ab8f58e_characterizeImage_855043_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:22 proc_lsst.multi:146 [INFO] found job 16912312 in provider astro proc_lsst.multi INFO: found job 16912312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:23 proc_lsst.multi:146 [INFO] found job 16912313 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912313 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:24 proc_lsst.multi:146 [INFO] found job 16912314 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912314 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:25 proc_lsst.multi:146 [INFO] found job 16912315 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912315 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:26 proc_lsst.multi:146 [INFO] found job 16912316 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912316 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:26 proc_lsst.multi:146 [INFO] found job 16912317 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:27 proc_lsst.multi:146 [INFO] found job 16912318 in provider escience_a40 proc_lsst.multi INFO: found job 16912318 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:28 proc_lsst.multi:146 [INFO] found job 16912319 in provider escience_a40 proc_lsst.multi INFO: found job 16912319 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:29 proc_lsst.multi:146 [INFO] found job 16912320 in provider escience_a40 proc_lsst.multi INFO: found job 16912320 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:31 proc_lsst.multi:146 [INFO] found job 16912321 in provider escience_a40 proc_lsst.multi INFO: found job 16912321 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:33 proc_lsst.multi:146 [INFO] found job 16912322 in provider escience_a40 proc_lsst.multi INFO: found job 16912322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 313 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 368b5a500fb3df9644db8783384bf0a6 with result from task 313 parsl.dataflow.dflow DEBUG: Task 313 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 314 has memoization hash 7eb4b5b67dafc987760761d0b6187492 parsl.dataflow.memoization INFO: Task 314 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 314 try 0 launched on executor multi with executor id 752 parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/5ccf1061-84a1-4826-9cc6-b9461d8c7fe9_calibrate_855042_31.stdout parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/5ccf1061-84a1-4826-9cc6-b9461d8c7fe9_calibrate_855042_31.stderr parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/879f1f6d-7610-48ad-970f-7d53090409df_characterizeImage_855042_31.stdout parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/879f1f6d-7610-48ad-970f-7d53090409df_characterizeImage_855042_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:36 proc_lsst.multi:146 [INFO] found job 16912323 in provider escience_a40 proc_lsst.multi INFO: found job 16912323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:37 proc_lsst.multi:146 [INFO] found job 16912324 in provider escience_a40 proc_lsst.multi INFO: found job 16912324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:38 proc_lsst.multi:146 [INFO] found job 16912325 in provider escience_a40 proc_lsst.multi INFO: found job 16912325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:39 proc_lsst.multi:146 [INFO] found job 16912326 in provider escience_a40 proc_lsst.multi INFO: found job 16912326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 391 try 1 failed parsl.dataflow.dflow ERROR: Task 391 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry a97b8755c078a2e49ccc0564639c1bb4 with result from task 391 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 392 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 392 try 0 failed parsl.dataflow.dflow INFO: Task 392 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/0a99dd08-56ff-45a0-9a09-0a5c607414ce_calibrate_855137_2.stdout parsl.dataflow.dflow INFO: Standard error for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/0a99dd08-56ff-45a0-9a09-0a5c607414ce_calibrate_855137_2.stderr parsl.dataflow.dflow INFO: Standard output for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/d7f79a9c-f7aa-42a3-84c1-ce23407ffc8d_characterizeImage_855137_2.stdout parsl.dataflow.dflow INFO: Standard error for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/d7f79a9c-f7aa-42a3-84c1-ce23407ffc8d_characterizeImage_855137_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 118 try 1 failed parsl.dataflow.dflow ERROR: Task 118 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 90171a04074af488fac4cfc595aa2314 with result from task 118 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 119 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 119 try 0 failed parsl.dataflow.dflow INFO: Task 119 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/aa1f71c3-09e8-472f-94ab-eaae0a588f03_calibrate_855042_2.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/aa1f71c3-09e8-472f-94ab-eaae0a588f03_calibrate_855042_2.stderr parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/f4ba88e8-9ec1-4d37-a2e0-cdd18c3b7056_characterizeImage_855042_2.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/f4ba88e8-9ec1-4d37-a2e0-cdd18c3b7056_characterizeImage_855042_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 919 try 1 failed parsl.dataflow.dflow ERROR: Task 919 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 11b95bba4e0daed4db9efaf1de5e401c with result from task 919 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 920 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 920 try 0 failed parsl.dataflow.dflow INFO: Task 920 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/8f31c3c5-cb95-462f-89f8-6cf544af9876_calibrate_855044_2.stdout parsl.dataflow.dflow INFO: Standard error for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/8f31c3c5-cb95-462f-89f8-6cf544af9876_calibrate_855044_2.stderr parsl.dataflow.dflow INFO: Standard output for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/80704541-bab4-4b60-9c40-45116b4567c1_characterizeImage_855044_2.stdout parsl.dataflow.dflow INFO: Standard error for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/80704541-bab4-4b60-9c40-45116b4567c1_characterizeImage_855044_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 874 try 1 failed parsl.dataflow.dflow ERROR: Task 874 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry d5ee3c2628c49d0e826ae7f7da60ef7f with result from task 874 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 875 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 875 try 0 failed parsl.dataflow.dflow INFO: Task 875 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/115a348d-ccc9-406a-a2c9-b14e5b27a014_calibrate_855136_2.stdout parsl.dataflow.dflow INFO: Standard error for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/115a348d-ccc9-406a-a2c9-b14e5b27a014_calibrate_855136_2.stderr parsl.dataflow.dflow INFO: Standard output for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/ec5235e9-6c9e-47d1-babb-4ec2b9224d0f_characterizeImage_855136_2.stdout parsl.dataflow.dflow INFO: Standard error for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/ec5235e9-6c9e-47d1-babb-4ec2b9224d0f_characterizeImage_855136_2.stderr parsl.dataflow.dflow INFO: Task 64 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 575c44743cc37efc60a08afd0388548d with result from task 64 parsl.dataflow.dflow DEBUG: Task 64 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 65 has memoization hash 566f3ddb68117498bf98105a3cd410d6 parsl.dataflow.memoization INFO: Task 65 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d2520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 65 try 0 launched on executor multi with executor id 753 parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/fc77bc70-3f39-4916-8f9f-1dac6f13929d_calibrate_855137_31.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/fc77bc70-3f39-4916-8f9f-1dac6f13929d_calibrate_855137_31.stderr parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/2c763691-9176-48d0-a610-fb07257ee159_characterizeImage_855137_31.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/2c763691-9176-48d0-a610-fb07257ee159_characterizeImage_855137_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:40 proc_lsst.multi:146 [INFO] found job 16912327 in provider escience_a40 proc_lsst.multi INFO: found job 16912327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:40 proc_lsst.multi:146 [INFO] found job 16912328 in provider escience_a40 proc_lsst.multi INFO: found job 16912328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:41 proc_lsst.multi:146 [INFO] found job 16912329 in provider escience_a40 proc_lsst.multi INFO: found job 16912329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 748 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0076747ec9709b221100d2c431ceba3a with result from task 748 parsl.dataflow.dflow DEBUG: Task 748 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 749 has memoization hash 0f34e23ac2c863527a0752d4dffde218 parsl.dataflow.memoization INFO: Task 749 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 749 try 0 launched on executor multi with executor id 754 parsl.dataflow.dflow INFO: Standard output for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/39c1c8c4-99e1-42f7-8bf7-c335fae945a1_calibrate_855136_31.stdout parsl.dataflow.dflow INFO: Standard error for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/39c1c8c4-99e1-42f7-8bf7-c335fae945a1_calibrate_855136_31.stderr parsl.dataflow.dflow INFO: Standard output for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/29a47264-15d8-4b28-a875-b43a0635e26f_characterizeImage_855136_31.stdout parsl.dataflow.dflow INFO: Standard error for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/29a47264-15d8-4b28-a875-b43a0635e26f_characterizeImage_855136_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:42 proc_lsst.multi:146 [INFO] found job 16912330 in provider escience_a40 proc_lsst.multi INFO: found job 16912330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 802 try 1 failed parsl.dataflow.dflow ERROR: Task 802 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry f194a5b3aed37b05e780c4da5d373ee1 with result from task 802 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 803 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 803 try 0 failed parsl.dataflow.dflow INFO: Task 803 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/8d9057aa-72a3-4aaa-92a2-c7c15c612531_calibrate_855138_2.stdout parsl.dataflow.dflow INFO: Standard error for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/8d9057aa-72a3-4aaa-92a2-c7c15c612531_calibrate_855138_2.stderr parsl.dataflow.dflow INFO: Standard output for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/476a5fdb-6792-4fc0-8b21-94ef66d94625_characterizeImage_855138_2.stdout parsl.dataflow.dflow INFO: Standard error for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/476a5fdb-6792-4fc0-8b21-94ef66d94625_characterizeImage_855138_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 595 try 1 failed parsl.dataflow.dflow ERROR: Task 595 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 6a5be1ea6724fa9b116cd160e08e79b3 with result from task 595 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 596 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 596 try 0 failed parsl.dataflow.dflow INFO: Task 596 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/cd3ee8ec-8e3e-4368-b5ee-685481ff0af9_calibrate_855043_2.stdout parsl.dataflow.dflow INFO: Standard error for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/cd3ee8ec-8e3e-4368-b5ee-685481ff0af9_calibrate_855043_2.stderr parsl.dataflow.dflow INFO: Standard output for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/69f251fe-c21e-46fd-adef-dd0cc1e49f52_characterizeImage_855043_2.stdout parsl.dataflow.dflow INFO: Standard error for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/69f251fe-c21e-46fd-adef-dd0cc1e49f52_characterizeImage_855043_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:43 proc_lsst.multi:146 [INFO] found job 16912331 in provider escience_a40 proc_lsst.multi INFO: found job 16912331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 586 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 918255a4edcba52134f51317bf1e21ee with result from task 586 parsl.dataflow.dflow DEBUG: Task 586 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 587 has memoization hash 463bee33dd3ba10d0e11bbb399065ca4 parsl.dataflow.memoization INFO: Task 587 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45743060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 587 try 0 launched on executor multi with executor id 755 parsl.dataflow.dflow INFO: Standard output for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/46bc6e0f-d910-49f4-9b22-1432a12bac1e_calibrate_855042_21.stdout parsl.dataflow.dflow INFO: Standard error for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/46bc6e0f-d910-49f4-9b22-1432a12bac1e_calibrate_855042_21.stderr parsl.dataflow.dflow INFO: Standard output for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/dc375654-3833-4292-bcce-6af3cad4d0eb_characterizeImage_855042_21.stdout parsl.dataflow.dflow INFO: Standard error for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/dc375654-3833-4292-bcce-6af3cad4d0eb_characterizeImage_855042_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:44 proc_lsst.multi:146 [INFO] found job 16912332 in provider escience_a40 proc_lsst.multi INFO: found job 16912332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:44 proc_lsst.multi:146 [INFO] found job 16912333 in provider escience_a40 proc_lsst.multi INFO: found job 16912333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 394 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f873ea2682fab05347002136b08d470d with result from task 394 parsl.dataflow.dflow DEBUG: Task 394 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 395 has memoization hash 68a3fe6c87fcfe259c899f4859fc5de4 parsl.dataflow.memoization INFO: Task 395 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4687bc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 395 try 0 launched on executor multi with executor id 756 parsl.dataflow.dflow INFO: Standard output for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/9bd703e0-ba48-46ba-98db-3b9193da9953_calibrate_855044_5.stdout parsl.dataflow.dflow INFO: Standard error for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/9bd703e0-ba48-46ba-98db-3b9193da9953_calibrate_855044_5.stderr parsl.dataflow.dflow INFO: Standard output for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/09ece147-f068-49e5-8b33-0a6860c9e8e9_characterizeImage_855044_5.stdout parsl.dataflow.dflow INFO: Standard error for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/09ece147-f068-49e5-8b33-0a6860c9e8e9_characterizeImage_855044_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:45 proc_lsst.multi:146 [INFO] found job 16912334 in provider escience_a40 proc_lsst.multi INFO: found job 16912334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 481 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5ba960e53fc80b81f5f6629e603cc85 with result from task 481 parsl.dataflow.dflow DEBUG: Task 481 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 482 has memoization hash 5e4b2b63d7af4bae6ca99e2502835e5d parsl.dataflow.memoization INFO: Task 482 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456cccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 482 try 0 launched on executor multi with executor id 757 parsl.dataflow.dflow INFO: Standard output for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/29135289-688d-44d2-9921-b773d886d0b5_calibrate_855044_31.stdout parsl.dataflow.dflow INFO: Standard error for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/29135289-688d-44d2-9921-b773d886d0b5_calibrate_855044_31.stderr parsl.dataflow.dflow INFO: Standard output for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/3d4cc1dd-b196-45bc-a21b-b20a76698007_characterizeImage_855044_31.stdout parsl.dataflow.dflow INFO: Standard error for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/3d4cc1dd-b196-45bc-a21b-b20a76698007_characterizeImage_855044_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:46 proc_lsst.multi:146 [INFO] found job 16912335 in provider escience_a40 proc_lsst.multi INFO: found job 16912335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:47 proc_lsst.multi:146 [INFO] found job 16912336 in provider escience_a40 proc_lsst.multi INFO: found job 16912336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 757 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d02daa61d8df9a1c47f43c766a1b1941 with result from task 757 parsl.dataflow.dflow DEBUG: Task 757 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 758 has memoization hash bf03690391893b3ad16f8ccd3d6545b0 parsl.dataflow.memoization INFO: Task 758 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d2c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 758 try 0 launched on executor multi with executor id 758 parsl.dataflow.dflow INFO: Standard output for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/514a49f3-60ba-44ed-afa6-af0c5d5416bd_calibrate_855044_10.stdout parsl.dataflow.dflow INFO: Standard error for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/514a49f3-60ba-44ed-afa6-af0c5d5416bd_calibrate_855044_10.stderr parsl.dataflow.dflow INFO: Standard output for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/c216f620-b82e-4520-ade0-c0f03b24f500_characterizeImage_855044_10.stdout parsl.dataflow.dflow INFO: Standard error for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/c216f620-b82e-4520-ade0-c0f03b24f500_characterizeImage_855044_10.stderr parsl.dataflow.dflow INFO: Task 709 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3516822fe2d377a91847afb9f42b05ee with result from task 709 parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:48 proc_lsst.multi:146 [INFO] found job 16912337 in provider escience_a40 proc_lsst.multi INFO: found job 16912337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow DEBUG: Task 709 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 710 has memoization hash c8f25b24782df584460d0f0052cb30f5 parsl.dataflow.memoization INFO: Task 710 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d0b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 710 try 0 launched on executor multi with executor id 759 parsl.dataflow.dflow INFO: Standard output for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/32499465-8270-4f57-b787-2511fa37d6f3_calibrate_855042_5.stdout parsl.dataflow.dflow INFO: Standard error for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/32499465-8270-4f57-b787-2511fa37d6f3_calibrate_855042_5.stderr parsl.dataflow.dflow INFO: Standard output for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/8ca427e1-013e-47bd-907f-6b8663f7cf67_characterizeImage_855042_5.stdout parsl.dataflow.dflow INFO: Standard error for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/8ca427e1-013e-47bd-907f-6b8663f7cf67_characterizeImage_855042_5.stderr parsl.dataflow.dflow INFO: Task 289 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6351b9b4d460a720eb496b0cbfbf077 with result from task 289 parsl.dataflow.dflow DEBUG: Task 289 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 290 has memoization hash 13411f8f5b303f0f9151aea814c68bd4 parsl.dataflow.memoization INFO: Task 290 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 290 try 0 launched on executor multi with executor id 760 parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/ed44fa25-0bae-4993-b20e-3273d0b9632e_calibrate_855138_13.stdout parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/ed44fa25-0bae-4993-b20e-3273d0b9632e_calibrate_855138_13.stderr parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/9c5712a2-b7f4-4b15-a542-0d04640e0e4a_characterizeImage_855138_13.stdout parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/9c5712a2-b7f4-4b15-a542-0d04640e0e4a_characterizeImage_855138_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:48 proc_lsst.multi:146 [INFO] found job 16912338 in provider checkpoint proc_lsst.multi INFO: found job 16912338 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:49 proc_lsst.multi:146 [INFO] found job 16912339 in provider checkpoint proc_lsst.multi INFO: found job 16912339 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 361 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49386d940c39af8bb9b8757344a6abb7 with result from task 361 parsl.dataflow.dflow DEBUG: Task 361 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 362 has memoization hash e85bfe198afee346003b2c845710c423 parsl.dataflow.memoization INFO: Task 362 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46879bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 362 try 0 launched on executor multi with executor id 761 parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/9124c5d5-6eaf-4584-a53b-984a2175b4f2_calibrate_855137_25.stdout parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/9124c5d5-6eaf-4584-a53b-984a2175b4f2_calibrate_855137_25.stderr parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/e5c6ea55-f107-43ab-9850-d92da9c9c562_characterizeImage_855137_25.stdout parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/e5c6ea55-f107-43ab-9850-d92da9c9c562_characterizeImage_855137_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:50 proc_lsst.multi:146 [INFO] found job 16912340 in provider checkpoint proc_lsst.multi INFO: found job 16912340 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:51 proc_lsst.multi:146 [INFO] found job 16912341 in provider checkpoint proc_lsst.multi INFO: found job 16912341 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 844 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72f51e751f2db88bb6de80a75c703dd8 with result from task 844 parsl.dataflow.dflow DEBUG: Task 844 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 845 has memoization hash 42584de13e0ba9c74cfadc2648ca97d7 parsl.dataflow.memoization INFO: Task 845 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456656c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 845 try 0 launched on executor multi with executor id 762 parsl.dataflow.dflow INFO: Standard output for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/09a864ff-b53b-45e8-9593-f1c4fadad042_calibrate_855043_40.stdout parsl.dataflow.dflow INFO: Standard error for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/09a864ff-b53b-45e8-9593-f1c4fadad042_calibrate_855043_40.stderr parsl.dataflow.dflow INFO: Standard output for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/e4b0b5da-f184-4492-bd2e-212e5511bbda_characterizeImage_855043_40.stdout parsl.dataflow.dflow INFO: Standard error for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/e4b0b5da-f184-4492-bd2e-212e5511bbda_characterizeImage_855043_40.stderr parsl.dataflow.dflow INFO: Task 385 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 26ac17feb0c9a1f2e6196a6537b6f3b9 with result from task 385 parsl.dataflow.dflow DEBUG: Task 385 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 386 has memoization hash 5d6ae31ed421517f5e87efdbcb4fcbec parsl.dataflow.memoization INFO: Task 386 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4687b1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 386 try 0 launched on executor multi with executor id 763 parsl.dataflow.dflow INFO: Standard output for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/d91c5031-e1ab-4162-af09-8af059d0fcaf_calibrate_855136_1.stdout parsl.dataflow.dflow INFO: Standard error for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/d91c5031-e1ab-4162-af09-8af059d0fcaf_calibrate_855136_1.stderr parsl.dataflow.dflow INFO: Standard output for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/5d4a2880-f84d-4e34-9f5e-e536350eba08_characterizeImage_855136_1.stdout parsl.dataflow.dflow INFO: Standard error for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/5d4a2880-f84d-4e34-9f5e-e536350eba08_characterizeImage_855136_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:52 proc_lsst.multi:146 [INFO] found job 16912342 in provider checkpoint proc_lsst.multi INFO: found job 16912342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 70 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b7e132571e3fc79cd71e18b821ebcc1 with result from task 70 parsl.dataflow.dflow DEBUG: Task 70 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 71 has memoization hash 6ffc6212f1ec24e34ea37cda4676e444 parsl.dataflow.memoization INFO: Task 71 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d2b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 71 try 0 launched on executor multi with executor id 764 parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/ee1c9923-93d0-4878-96bc-d8aaa67f64f5_calibrate_855137_4.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/ee1c9923-93d0-4878-96bc-d8aaa67f64f5_calibrate_855137_4.stderr parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/498ab39c-a1c9-4391-b8a0-aa2ce9b1733a_characterizeImage_855137_4.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/498ab39c-a1c9-4391-b8a0-aa2ce9b1733a_characterizeImage_855137_4.stderr parsl.dataflow.dflow INFO: Task 583 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a329a61a85043143edb37ebe69966713 with result from task 583 parsl.dataflow.dflow DEBUG: Task 583 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 584 has memoization hash 424e677337099e6b9560db704a53e67c parsl.dataflow.memoization INFO: Task 584 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45743380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 584 try 0 launched on executor multi with executor id 765 parsl.dataflow.dflow INFO: Standard output for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/b1d9c744-6fe0-4880-a5b5-e4e5ebb1e041_calibrate_855042_57.stdout parsl.dataflow.dflow INFO: Standard error for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/b1d9c744-6fe0-4880-a5b5-e4e5ebb1e041_calibrate_855042_57.stderr parsl.dataflow.dflow INFO: Standard output for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/7900b214-4a80-4deb-9c2d-7b0fa7cff3fb_characterizeImage_855042_57.stdout parsl.dataflow.dflow INFO: Standard error for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/7900b214-4a80-4deb-9c2d-7b0fa7cff3fb_characterizeImage_855042_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:53 proc_lsst.multi:146 [INFO] found job 16912343 in provider checkpoint proc_lsst.multi INFO: found job 16912343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:53 proc_lsst.multi:146 [INFO] found job 16912344 in provider checkpoint proc_lsst.multi INFO: found job 16912344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 883 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b973a94ef7b766f9cbaa1562ff69e0b3 with result from task 883 parsl.dataflow.dflow DEBUG: Task 883 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 884 has memoization hash 0aea7554b4fa411232e68dcc5ac15d65 parsl.dataflow.memoization INFO: Task 884 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45667880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 884 try 0 launched on executor multi with executor id 766 parsl.dataflow.dflow INFO: Standard output for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/70a3087e-ea01-4237-9720-fd2c311e4cdd_calibrate_855138_3.stdout parsl.dataflow.dflow INFO: Standard error for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/70a3087e-ea01-4237-9720-fd2c311e4cdd_calibrate_855138_3.stderr parsl.dataflow.dflow INFO: Standard output for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/daaf5bb5-4445-4596-8c3a-1c59b07acc2f_characterizeImage_855138_3.stdout parsl.dataflow.dflow INFO: Standard error for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/daaf5bb5-4445-4596-8c3a-1c59b07acc2f_characterizeImage_855138_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:54 proc_lsst.multi:146 [INFO] found job 16912345 in provider checkpoint proc_lsst.multi INFO: found job 16912345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 880 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca94007e17e771b8d0f08a47d801081e with result from task 880 parsl.dataflow.dflow DEBUG: Task 880 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 881 has memoization hash bcdad3f205fd2ba6ad9eeeca412bba20 parsl.dataflow.memoization INFO: Task 881 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45667560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 881 try 0 launched on executor multi with executor id 767 parsl.dataflow.dflow INFO: Standard output for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/7f60b30c-5fcb-48d8-9300-7e7d00e94df4_calibrate_855043_46.stdout parsl.dataflow.dflow INFO: Standard error for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/7f60b30c-5fcb-48d8-9300-7e7d00e94df4_calibrate_855043_46.stderr parsl.dataflow.dflow INFO: Standard output for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/39ec4d97-85f2-42b4-814d-61f21658e2fd_characterizeImage_855043_46.stdout parsl.dataflow.dflow INFO: Standard error for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/39ec4d97-85f2-42b4-814d-61f21658e2fd_characterizeImage_855043_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:55 proc_lsst.multi:146 [INFO] found job 16912346 in provider checkpoint proc_lsst.multi INFO: found job 16912346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 409 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b20a6050b61074153fc65514209be77d with result from task 409 parsl.dataflow.dflow DEBUG: Task 409 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 410 has memoization hash fcb51f98daa6904df8b803a8f4c1180f parsl.dataflow.memoization INFO: Task 410 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a7380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 410 try 0 launched on executor multi with executor id 768 parsl.dataflow.dflow INFO: Standard output for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/4b6f9edd-8931-451d-8258-c503abeb6f91_calibrate_855043_48.stdout parsl.dataflow.dflow INFO: Standard error for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/4b6f9edd-8931-451d-8258-c503abeb6f91_calibrate_855043_48.stderr parsl.dataflow.dflow INFO: Standard output for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/b4179868-7eca-4d3f-a9cf-cb6b3cd82fdf_characterizeImage_855043_48.stdout parsl.dataflow.dflow INFO: Standard error for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/b4179868-7eca-4d3f-a9cf-cb6b3cd82fdf_characterizeImage_855043_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:56 proc_lsst.multi:146 [INFO] found job 16912347 in provider checkpoint proc_lsst.multi INFO: found job 16912347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 742 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f33b90f3a9350a469cba1d9f9d03ffe with result from task 742 parsl.dataflow.dflow DEBUG: Task 742 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 743 has memoization hash e2f700d10a06e7d3636f708002e00b11 parsl.dataflow.memoization INFO: Task 743 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 743 try 0 launched on executor multi with executor id 769 parsl.dataflow.dflow INFO: Standard output for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/de6beae1-c85c-44db-98f4-3ac0e345c834_calibrate_855042_50.stdout parsl.dataflow.dflow INFO: Standard error for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/de6beae1-c85c-44db-98f4-3ac0e345c834_calibrate_855042_50.stderr parsl.dataflow.dflow INFO: Standard output for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/fba365d7-440a-40a9-bf2f-2b1f25924869_characterizeImage_855042_50.stdout parsl.dataflow.dflow INFO: Standard error for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/fba365d7-440a-40a9-bf2f-2b1f25924869_characterizeImage_855042_50.stderr parsl.dataflow.dflow INFO: Task 775 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7d557705dc70e9273852840735d842c with result from task 775 parsl.dataflow.dflow DEBUG: Task 775 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 776 has memoization hash 558263fe67fae5a535b0ba23e584c1e4 parsl.dataflow.memoization INFO: Task 776 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560c0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 776 try 0 launched on executor multi with executor id 770 parsl.dataflow.dflow INFO: Standard output for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/879c90aa-9458-48c9-8ad7-205ea964d056_calibrate_855138_38.stdout parsl.dataflow.dflow INFO: Standard error for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/879c90aa-9458-48c9-8ad7-205ea964d056_calibrate_855138_38.stderr parsl.dataflow.dflow INFO: Standard output for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/143b6809-5c04-4ed0-8eca-437097873506_characterizeImage_855138_38.stdout parsl.dataflow.dflow INFO: Standard error for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/143b6809-5c04-4ed0-8eca-437097873506_characterizeImage_855138_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:57 proc_lsst.multi:146 [INFO] found job 16912348 in provider checkpoint proc_lsst.multi INFO: found job 16912348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 769 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9425327743364f7cdf475b1a2d18afb1 with result from task 769 parsl.dataflow.dflow DEBUG: Task 769 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 770 has memoization hash 482ccc606cd1ffdbc5e9a98459c92c39 parsl.dataflow.memoization INFO: Task 770 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d39c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 770 try 0 launched on executor multi with executor id 771 parsl.dataflow.dflow INFO: Standard output for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/bc063c4f-063d-4481-a1e0-0212543d9629_calibrate_855137_1.stdout parsl.dataflow.dflow INFO: Standard error for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/bc063c4f-063d-4481-a1e0-0212543d9629_calibrate_855137_1.stderr parsl.dataflow.dflow INFO: Standard output for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/1a3e5edb-849f-4780-9395-b6795a89e8f1_characterizeImage_855137_1.stdout parsl.dataflow.dflow INFO: Standard error for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/1a3e5edb-849f-4780-9395-b6795a89e8f1_characterizeImage_855137_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:57 proc_lsst.multi:146 [INFO] found job 16912349 in provider checkpoint proc_lsst.multi INFO: found job 16912349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 445 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd6c834262be70d6be32ab34bab3681e with result from task 445 parsl.dataflow.dflow DEBUG: Task 445 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 446 has memoization hash f8b7c8accc7eff676b01b3ea68406bf6 parsl.dataflow.memoization INFO: Task 446 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a6840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 446 try 0 launched on executor multi with executor id 772 parsl.dataflow.dflow INFO: Standard output for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/5f50344f-c795-4405-a281-40c8d875ad80_calibrate_855043_1.stdout parsl.dataflow.dflow INFO: Standard error for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/5f50344f-c795-4405-a281-40c8d875ad80_calibrate_855043_1.stderr parsl.dataflow.dflow INFO: Standard output for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/8cdb98f6-cf32-47bf-905d-609b84a201a5_characterizeImage_855043_1.stdout parsl.dataflow.dflow INFO: Standard error for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/8cdb98f6-cf32-47bf-905d-609b84a201a5_characterizeImage_855043_1.stderr parsl.dataflow.dflow INFO: Task 427 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4332d370b193ab64500c63fa1a356635 with result from task 427 parsl.dataflow.dflow DEBUG: Task 427 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 428 has memoization hash 49a703e1f42f140b616240290c54eadf parsl.dataflow.memoization INFO: Task 428 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a5620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 428 try 0 launched on executor multi with executor id 773 parsl.dataflow.dflow INFO: Standard output for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/bd6ab114-a354-4d11-82ed-a64d9fcc4bae_calibrate_855136_25.stdout parsl.dataflow.dflow INFO: Standard error for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/bd6ab114-a354-4d11-82ed-a64d9fcc4bae_calibrate_855136_25.stderr parsl.dataflow.dflow INFO: Standard output for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/2465daeb-2c94-470c-af3c-eaed21ce6fa4_characterizeImage_855136_25.stdout parsl.dataflow.dflow INFO: Standard error for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/2465daeb-2c94-470c-af3c-eaed21ce6fa4_characterizeImage_855136_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:58 proc_lsst.multi:146 [INFO] found job 16912350 in provider checkpoint proc_lsst.multi INFO: found job 16912350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:17:59 proc_lsst.multi:146 [INFO] found job 16912351 in provider checkpoint proc_lsst.multi INFO: found job 16912351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 130 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4bc04a5cd170a517d8ba58deeea89adc with result from task 130 parsl.dataflow.dflow DEBUG: Task 130 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 131 has memoization hash 0127ca55dc8652c1c3020e148bdb7bdc parsl.dataflow.memoization INFO: Task 131 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f6a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 131 try 0 launched on executor multi with executor id 774 parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/27af0d40-1608-43c3-b008-702315fc5bb3_calibrate_855042_3.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/27af0d40-1608-43c3-b008-702315fc5bb3_calibrate_855042_3.stderr parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/44388d39-c1f5-4643-8410-46500f9c9f82_characterizeImage_855042_3.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/44388d39-c1f5-4643-8410-46500f9c9f82_characterizeImage_855042_3.stderr parsl.dataflow.dflow INFO: Task 82 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 88dc7bb497c36290e27d79d0fb453e8f with result from task 82 parsl.dataflow.dflow DEBUG: Task 82 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 83 has memoization hash 494920ece182dfc0b7a1e6e082e495a7 parsl.dataflow.memoization INFO: Task 83 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d37e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 83 try 0 launched on executor multi with executor id 775 parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/d2ee6f41-45d3-4b59-ace2-b0d84370fe4d_calibrate_855138_8.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/d2ee6f41-45d3-4b59-ace2-b0d84370fe4d_calibrate_855138_8.stderr parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/9c3a29e5-0fba-46e2-aa95-5515ba146a65_characterizeImage_855138_8.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/9c3a29e5-0fba-46e2-aa95-5515ba146a65_characterizeImage_855138_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:18:00 proc_lsst.multi:146 [INFO] found job 16912352 in provider checkpoint proc_lsst.multi INFO: found job 16912352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:18:01 proc_lsst.multi:146 [INFO] found job 16912353 in provider checkpoint proc_lsst.multi INFO: found job 16912353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 40 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5cd2d395835751e630b18c6d4457f80 with result from task 40 parsl.dataflow.dflow DEBUG: Task 40 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 41 has memoization hash be46eaad2ae3625fab2cb12d24746b5e parsl.dataflow.memoization INFO: Task 41 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d0e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 41 try 0 launched on executor multi with executor id 776 parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/9a4a1baa-c422-450a-9a4b-a9a99c30b564_calibrate_855042_62.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/9a4a1baa-c422-450a-9a4b-a9a99c30b564_calibrate_855042_62.stderr parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/d948cbcf-5a02-4ab5-9cf1-c3a838990325_characterizeImage_855042_62.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/d948cbcf-5a02-4ab5-9cf1-c3a838990325_characterizeImage_855042_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:18:02 proc_lsst.multi:146 [INFO] found job 16912354 in provider checkpoint proc_lsst.multi INFO: found job 16912354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 283 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8df2da1b1a572a7a4056ff897a99cda with result from task 283 parsl.dataflow.dflow DEBUG: Task 283 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 284 has memoization hash 4a86ef2bd71b7ba61f90ffef903aa9e1 parsl.dataflow.memoization INFO: Task 284 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 284 try 0 launched on executor multi with executor id 777 parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/724cbfce-0c03-4f11-9b40-f7d2cb1e45e6_calibrate_855043_36.stdout parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/724cbfce-0c03-4f11-9b40-f7d2cb1e45e6_calibrate_855043_36.stderr parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/4541dd72-ab0b-4cad-9e69-75528dc83160_characterizeImage_855043_36.stdout parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/4541dd72-ab0b-4cad-9e69-75528dc83160_characterizeImage_855043_36.stderr parsl.dataflow.dflow INFO: Task 580 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f951cb7270da796f820346ac089394c8 with result from task 580 parsl.dataflow.dflow DEBUG: Task 580 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 581 has memoization hash d4059b5e63cc9c750eabb3c4aa917a97 parsl.dataflow.memoization INFO: Task 581 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45743740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 581 try 0 launched on executor multi with executor id 778 parsl.dataflow.dflow INFO: Standard output for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/da386ffd-8813-49aa-b365-39f2edc245b1_calibrate_855043_58.stdout parsl.dataflow.dflow INFO: Standard error for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/da386ffd-8813-49aa-b365-39f2edc245b1_calibrate_855043_58.stderr parsl.dataflow.dflow INFO: Standard output for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/1a51505a-2a77-4d9e-a240-cc02b83866d1_characterizeImage_855043_58.stdout parsl.dataflow.dflow INFO: Standard error for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/1a51505a-2a77-4d9e-a240-cc02b83866d1_characterizeImage_855043_58.stderr parsl.dataflow.dflow INFO: Task 700 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6cf73c159371afcc10b888baff4fecc5 with result from task 700 parsl.dataflow.dflow DEBUG: Task 700 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 701 has memoization hash 47b0e9abcddf32f5fc465860adfbc4b2 parsl.dataflow.memoization INFO: Task 701 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d0220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 701 try 0 launched on executor multi with executor id 779 parsl.dataflow.dflow INFO: Standard output for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/9ee55245-ff5b-41bf-b1a4-7066db455a01_calibrate_855043_45.stdout parsl.dataflow.dflow INFO: Standard error for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/9ee55245-ff5b-41bf-b1a4-7066db455a01_calibrate_855043_45.stderr parsl.dataflow.dflow INFO: Standard output for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/269e9f7c-2de1-4307-b32f-4a415026c400_characterizeImage_855043_45.stdout parsl.dataflow.dflow INFO: Standard error for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/269e9f7c-2de1-4307-b32f-4a415026c400_characterizeImage_855043_45.stderr parsl.dataflow.dflow INFO: Task 19 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9db8a832c5466615d7c1b37af88a27e with result from task 19 parsl.dataflow.dflow DEBUG: Task 19 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 20 has memoization hash 6c5b9d5b74e90f54b2dd52473c4dd3fd parsl.dataflow.memoization INFO: Task 20 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467b1bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 20 try 0 launched on executor multi with executor id 780 parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/f9c8778d-75cd-4aa9-a668-a3562a6fa456_calibrate_855044_3.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/f9c8778d-75cd-4aa9-a668-a3562a6fa456_calibrate_855044_3.stderr parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/f0c42f63-1720-43d6-975a-b877f9843538_characterizeImage_855044_3.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/f0c42f63-1720-43d6-975a-b877f9843538_characterizeImage_855044_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:18:02 proc_lsst.multi:146 [INFO] found job 16912355 in provider checkpoint proc_lsst.multi INFO: found job 16912355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 322 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c98dd3f8e2816187e679f10c2911d0d with result from task 322 parsl.dataflow.dflow DEBUG: Task 322 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 323 has memoization hash 215e2f19ab8d82fa0bcd1799c1a5f5ac parsl.dataflow.memoization INFO: Task 323 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 323 try 0 launched on executor multi with executor id 781 parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/dd174b2e-e289-4f22-acd6-73ee59553a04_calibrate_855042_28.stdout parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/dd174b2e-e289-4f22-acd6-73ee59553a04_calibrate_855042_28.stderr parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/296c1234-f66c-4750-a817-eb5c3b24d7da_characterizeImage_855042_28.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/296c1234-f66c-4750-a817-eb5c3b24d7da_characterizeImage_855042_28.stderr parsl.dataflow.dflow INFO: Task 760 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e7743679a34288a3d50fe35305d3b76 with result from task 760 parsl.dataflow.dflow DEBUG: Task 760 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 761 has memoization hash 4d759ab6550c6f8b2ec1fee021e34022 parsl.dataflow.memoization INFO: Task 761 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d3100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 761 try 0 launched on executor multi with executor id 782 parsl.dataflow.dflow INFO: Standard output for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/309b3cd3-866a-43fa-aa5c-eca8039b986e_calibrate_855043_21.stdout parsl.dataflow.dflow INFO: Standard error for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/309b3cd3-866a-43fa-aa5c-eca8039b986e_calibrate_855043_21.stderr parsl.dataflow.dflow INFO: Standard output for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/6a8ffdf0-a4fc-44dd-8e7b-1cf29dadc7fe_characterizeImage_855043_21.stdout parsl.dataflow.dflow INFO: Standard error for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/6a8ffdf0-a4fc-44dd-8e7b-1cf29dadc7fe_characterizeImage_855043_21.stderr parsl.dataflow.dflow INFO: Task 565 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c143306785900dbb7a1e3c7716bfec0a with result from task 565 parsl.dataflow.dflow DEBUG: Task 565 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 566 has memoization hash de77b89c5ded6d97c9d15eb2b0eed6b0 parsl.dataflow.memoization INFO: Task 566 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45702c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 566 try 0 launched on executor multi with executor id 783 parsl.dataflow.dflow INFO: Standard output for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/8d5b2d58-6bce-43b5-b8e3-a97556aef937_calibrate_855136_7.stdout parsl.dataflow.dflow INFO: Standard error for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/8d5b2d58-6bce-43b5-b8e3-a97556aef937_calibrate_855136_7.stderr parsl.dataflow.dflow INFO: Standard output for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/c0895e9e-fd27-4c41-a41b-c6ea86b0efa5_characterizeImage_855136_7.stdout parsl.dataflow.dflow INFO: Standard error for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/c0895e9e-fd27-4c41-a41b-c6ea86b0efa5_characterizeImage_855136_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:18:03 proc_lsst.multi:146 [INFO] found job 16912356 in provider checkpoint proc_lsst.multi INFO: found job 16912356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 895 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d02c621a1ac36c4b3989846cce05d94 with result from task 895 parsl.dataflow.dflow DEBUG: Task 895 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 896 has memoization hash 85cef6554c6e53b4c5ff6eb22e94b4db parsl.dataflow.memoization INFO: Task 896 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45498040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 896 try 0 launched on executor multi with executor id 784 parsl.dataflow.dflow INFO: Standard output for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/f0e1657f-44c0-46a5-96c9-be04538ceb0f_calibrate_855042_25.stdout parsl.dataflow.dflow INFO: Standard error for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/f0e1657f-44c0-46a5-96c9-be04538ceb0f_calibrate_855042_25.stderr parsl.dataflow.dflow INFO: Standard output for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/bd876c83-3e4f-43e8-8b6d-1e36b03745fa_characterizeImage_855042_25.stdout parsl.dataflow.dflow INFO: Standard error for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/bd876c83-3e4f-43e8-8b6d-1e36b03745fa_characterizeImage_855042_25.stderr parsl.dataflow.dflow INFO: Task 37 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b1d845a3183f8c3439435972138c0905 with result from task 37 parsl.dataflow.dflow DEBUG: Task 37 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 38 has memoization hash c04088ad2243d7d54957386e1d4bf2e6 parsl.dataflow.memoization INFO: Task 38 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d0a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 38 try 0 launched on executor multi with executor id 785 parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/0dab76d9-b659-491b-922d-08f0dc77e48a_calibrate_855138_42.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/0dab76d9-b659-491b-922d-08f0dc77e48a_calibrate_855138_42.stderr parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/fbbb8cf4-a5ed-4e84-8876-4b6e31030e57_characterizeImage_855138_42.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/fbbb8cf4-a5ed-4e84-8876-4b6e31030e57_characterizeImage_855138_42.stderr parsl.dataflow.dflow INFO: Task 280 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a117625c4e901a2c91224ee79c950d9 with result from task 280 parsl.dataflow.dflow DEBUG: Task 280 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 281 has memoization hash 86f5b6fa80ca52666f0f46fa83fa23b1 parsl.dataflow.memoization INFO: Task 281 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 281 try 0 launched on executor multi with executor id 786 parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/1744c0c6-02db-4619-9dd1-3400c3bb866f_calibrate_855044_1.stdout parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/1744c0c6-02db-4619-9dd1-3400c3bb866f_calibrate_855044_1.stderr parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/1f2d1f4d-eb13-4ac8-86c7-7ab5ed480357_characterizeImage_855044_1.stdout parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/1f2d1f4d-eb13-4ac8-86c7-7ab5ed480357_characterizeImage_855044_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:18:04 proc_lsst.multi:146 [INFO] found job 16912359 in provider checkpoint proc_lsst.multi INFO: found job 16912359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 658 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7363a7271252e9e77b0f179eeaaaddb8 with result from task 658 parsl.dataflow.dflow DEBUG: Task 658 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 659 has memoization hash 80725b2969b3eee1006e5a9af3b08a10 parsl.dataflow.memoization INFO: Task 659 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 659 try 0 launched on executor multi with executor id 787 parsl.dataflow.dflow INFO: Standard output for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/26f6e728-5333-405c-9ce5-dc71236d69c0_calibrate_855042_36.stdout parsl.dataflow.dflow INFO: Standard error for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/26f6e728-5333-405c-9ce5-dc71236d69c0_calibrate_855042_36.stderr parsl.dataflow.dflow INFO: Standard output for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/4cdb4b4a-5888-42f3-a19b-ca8672765ff7_characterizeImage_855042_36.stdout parsl.dataflow.dflow INFO: Standard error for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/4cdb4b4a-5888-42f3-a19b-ca8672765ff7_characterizeImage_855042_36.stderr parsl.dataflow.dflow INFO: Task 7 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2d9112c3c8e05ec8b561d32911f3d6b with result from task 7 parsl.dataflow.dflow DEBUG: Task 7 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 8 has memoization hash 431c8786028091c30097524d1eaaa862 parsl.dataflow.memoization INFO: Task 8 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467b1260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 8 try 0 launched on executor multi with executor id 788 parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/07a68777-e012-4485-a851-a72643564d0b_calibrate_855044_41.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/07a68777-e012-4485-a851-a72643564d0b_calibrate_855044_41.stderr parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/79bf686e-fa33-461d-b1ea-0bed46f8f9b4_characterizeImage_855044_41.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/79bf686e-fa33-461d-b1ea-0bed46f8f9b4_characterizeImage_855044_41.stderr parsl.dataflow.dflow INFO: Task 100 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e6927f3529c5c127f71f6a393f5cd77 with result from task 100 parsl.dataflow.dflow DEBUG: Task 100 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 101 has memoization hash 352a6bcabf2c88b2c2a358167558afee parsl.dataflow.memoization INFO: Task 101 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f4ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 101 try 0 launched on executor multi with executor id 789 parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/0d931dc0-3286-4301-aef0-20e858f502ee_calibrate_855042_49.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/0d931dc0-3286-4301-aef0-20e858f502ee_calibrate_855042_49.stderr parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/ccc10e65-bd90-4813-ba42-c128f8516bbd_characterizeImage_855042_49.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/ccc10e65-bd90-4813-ba42-c128f8516bbd_characterizeImage_855042_49.stderr parsl.dataflow.dflow INFO: Task 55 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d071f37d13ed32c008a7064673b7d58c with result from task 55 parsl.dataflow.dflow DEBUG: Task 55 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 56 has memoization hash de9049970472bbb4ec1d8a10032f40ed parsl.dataflow.memoization INFO: Task 56 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d1bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 56 try 0 launched on executor multi with executor id 790 parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/d2c6f2f9-f3d0-4295-9171-05793d42a4d7_calibrate_855044_8.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/d2c6f2f9-f3d0-4295-9171-05793d42a4d7_calibrate_855044_8.stderr parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/9301d035-4d33-4249-aba1-7e5b01ae9e91_characterizeImage_855044_8.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/9301d035-4d33-4249-aba1-7e5b01ae9e91_characterizeImage_855044_8.stderr parsl.dataflow.dflow INFO: Task 163 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff9690ffe4be9be9d52fdf1209e90270 with result from task 163 parsl.dataflow.dflow DEBUG: Task 163 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 164 has memoization hash b2f1d90a5866200c59f5eea0d5f73b37 parsl.dataflow.memoization INFO: Task 164 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46814cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 164 try 0 launched on executor multi with executor id 791 parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/6401e3ce-0afd-4b57-8a70-b7e10a6e50d5_calibrate_855042_35.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/6401e3ce-0afd-4b57-8a70-b7e10a6e50d5_calibrate_855042_35.stderr parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/44527b48-1ff5-4e77-aa8c-b87d9b549060_characterizeImage_855042_35.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/44527b48-1ff5-4e77-aa8c-b87d9b549060_characterizeImage_855042_35.stderr parsl.dataflow.dflow INFO: Task 370 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a7f421a68c2a2526a1512ebaecec823e with result from task 370 parsl.dataflow.dflow DEBUG: Task 370 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 371 has memoization hash 11a2c733b159598ab1fad404a7505aa6 parsl.dataflow.memoization INFO: Task 371 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4687a520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 371 try 0 launched on executor multi with executor id 792 parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/94f54cfe-66c6-414c-9416-e6156e3214a2_calibrate_855137_3.stdout parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/94f54cfe-66c6-414c-9416-e6156e3214a2_calibrate_855137_3.stderr parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/d105a0b1-b098-4508-8693-9806986cfac7_characterizeImage_855137_3.stdout parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/d105a0b1-b098-4508-8693-9806986cfac7_characterizeImage_855137_3.stderr parsl.dataflow.dflow INFO: Task 406 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aab2e39a471790184c4908d49de76a37 with result from task 406 parsl.dataflow.dflow DEBUG: Task 406 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 407 has memoization hash a7d24710b49044cdd762e554e84c34fc parsl.dataflow.memoization INFO: Task 407 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a76a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 407 try 0 launched on executor multi with executor id 793 parsl.dataflow.dflow INFO: Standard output for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/2b45802e-5dd6-456d-83d0-eaf775bf7b50_calibrate_855137_45.stdout parsl.dataflow.dflow INFO: Standard error for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/2b45802e-5dd6-456d-83d0-eaf775bf7b50_calibrate_855137_45.stderr parsl.dataflow.dflow INFO: Standard output for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/f93ccfa3-d707-4b23-ab51-6ccbad917d52_characterizeImage_855137_45.stdout parsl.dataflow.dflow INFO: Standard error for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/f93ccfa3-d707-4b23-ab51-6ccbad917d52_characterizeImage_855137_45.stderr parsl.dataflow.dflow INFO: Task 337 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86e7412b47ae07a4f9c50c4b0f5eb658 with result from task 337 parsl.dataflow.dflow DEBUG: Task 337 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 338 has memoization hash cf6cfa249a1ef4867e9110913ebbb141 parsl.dataflow.memoization INFO: Task 338 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa468782c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 338 try 0 launched on executor multi with executor id 794 parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/f5132bfe-afe5-4892-9de2-5098a0ab0a16_calibrate_855043_5.stdout parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/f5132bfe-afe5-4892-9de2-5098a0ab0a16_calibrate_855043_5.stderr parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/071f77e6-ab22-4547-9959-6c4d543c20d5_characterizeImage_855043_5.stdout parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/071f77e6-ab22-4547-9959-6c4d543c20d5_characterizeImage_855043_5.stderr parsl.dataflow.dflow INFO: Task 772 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f761cef53898b1e8f335be29d3d3741a with result from task 772 parsl.dataflow.dflow DEBUG: Task 772 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 773 has memoization hash 41f428cd545b3635e8ce3a2d10b541c7 parsl.dataflow.memoization INFO: Task 773 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d3d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 773 try 0 launched on executor multi with executor id 795 parsl.dataflow.dflow INFO: Standard output for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/c1f7ecec-0123-497a-a8ce-01e2bceacb1a_calibrate_855043_37.stdout parsl.dataflow.dflow INFO: Standard error for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/c1f7ecec-0123-497a-a8ce-01e2bceacb1a_calibrate_855043_37.stderr parsl.dataflow.dflow INFO: Standard output for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/6da68ef4-c372-4ca2-81a0-8df5a06b9fd0_characterizeImage_855043_37.stdout parsl.dataflow.dflow INFO: Standard error for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/6da68ef4-c372-4ca2-81a0-8df5a06b9fd0_characterizeImage_855043_37.stderr parsl.dataflow.dflow INFO: Task 145 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0569beb8e6be32cb8ef4189757ed372a with result from task 145 parsl.dataflow.dflow DEBUG: Task 145 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 146 has memoization hash eab2957b36e32d07261a5bf4c2d68728 parsl.dataflow.memoization INFO: Task 146 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f79c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 146 try 0 launched on executor multi with executor id 796 parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/b98323b9-6f9b-4c14-bfaf-6e6e24fbb528_calibrate_855042_45.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/b98323b9-6f9b-4c14-bfaf-6e6e24fbb528_calibrate_855042_45.stderr parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/f8887070-aa46-408d-9b36-e3a52a4f5430_characterizeImage_855042_45.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/f8887070-aa46-408d-9b36-e3a52a4f5430_characterizeImage_855042_45.stderr parsl.dataflow.dflow INFO: Task 91 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 75c64e3911097d98d18c83d5ff9534ae with result from task 91 parsl.dataflow.dflow DEBUG: Task 91 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 92 has memoization hash 571c563f9d0dc517510e8eceb61ee335 parsl.dataflow.memoization INFO: Task 92 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f4180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 92 try 0 launched on executor multi with executor id 797 parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/39e58f0f-a79d-46bb-a573-c1a615bc8663_calibrate_855043_19.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/39e58f0f-a79d-46bb-a573-c1a615bc8663_calibrate_855043_19.stderr parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/5a73d06e-924a-40e1-a339-a4d4db7617bb_characterizeImage_855043_19.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/5a73d06e-924a-40e1-a339-a4d4db7617bb_characterizeImage_855043_19.stderr parsl.dataflow.dflow INFO: Task 442 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ab4494e1f109b7c79590b2d5bd66388 with result from task 442 parsl.dataflow.dflow DEBUG: Task 442 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 443 has memoization hash 1e8edfae81bc13a2e36c5e9786c65684 parsl.dataflow.memoization INFO: Task 443 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a6520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 443 try 0 launched on executor multi with executor id 798 parsl.dataflow.dflow INFO: Standard output for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/52e64ecc-70d6-47b4-a2ea-4417e5340936_calibrate_855138_1.stdout parsl.dataflow.dflow INFO: Standard error for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/52e64ecc-70d6-47b4-a2ea-4417e5340936_calibrate_855138_1.stderr parsl.dataflow.dflow INFO: Standard output for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/2721a225-c78f-4430-868d-65e1fa3907dd_characterizeImage_855138_1.stdout parsl.dataflow.dflow INFO: Standard error for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/2721a225-c78f-4430-868d-65e1fa3907dd_characterizeImage_855138_1.stderr parsl.dataflow.dflow INFO: Task 820 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7bba88759cfc233438c595769bbad1ed with result from task 820 parsl.dataflow.dflow DEBUG: Task 820 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 821 has memoization hash 406841e6406f277b7594f7d855f0905f parsl.dataflow.memoization INFO: Task 821 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 821 try 0 launched on executor multi with executor id 799 parsl.dataflow.dflow INFO: Standard output for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/2fe5a49e-48ac-452a-81b1-b4c933a85941_calibrate_855042_58.stdout parsl.dataflow.dflow INFO: Standard error for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/2fe5a49e-48ac-452a-81b1-b4c933a85941_calibrate_855042_58.stderr parsl.dataflow.dflow INFO: Standard output for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/75b934e6-c91a-42e4-8d3a-502a30b56220_characterizeImage_855042_58.stdout parsl.dataflow.dflow INFO: Standard error for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/75b934e6-c91a-42e4-8d3a-502a30b56220_characterizeImage_855042_58.stderr parsl.dataflow.dflow INFO: Task 793 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 340a0cb1a57163152c5b89daee2456bb with result from task 793 parsl.dataflow.dflow DEBUG: Task 793 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 794 has memoization hash f6cbe34e1c5557c879b1c17b6edef7b0 parsl.dataflow.memoization INFO: Task 794 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 794 try 0 launched on executor multi with executor id 800 parsl.dataflow.dflow INFO: Standard output for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/efdd8dde-237a-4786-ba7d-e487ef81af4c_calibrate_855043_49.stdout parsl.dataflow.dflow INFO: Standard error for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/efdd8dde-237a-4786-ba7d-e487ef81af4c_calibrate_855043_49.stderr parsl.dataflow.dflow INFO: Standard output for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/da3b2a11-677c-441d-b317-67909a54f1de_characterizeImage_855043_49.stdout parsl.dataflow.dflow INFO: Standard error for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/da3b2a11-677c-441d-b317-67909a54f1de_characterizeImage_855043_49.stderr parsl.dataflow.dflow INFO: Task 433 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b86fe75cff5bd5dac6b3b6ad1729332 with result from task 433 parsl.dataflow.dflow DEBUG: Task 433 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 434 has memoization hash f0ab61dd64729c41f1991d381d3a8f8c parsl.dataflow.memoization INFO: Task 434 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a4f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 434 try 0 launched on executor multi with executor id 801 parsl.dataflow.dflow INFO: Standard output for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/1b2819fd-ea0d-4b23-bd34-2dca58de5e0a_calibrate_855042_27.stdout parsl.dataflow.dflow INFO: Standard error for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/1b2819fd-ea0d-4b23-bd34-2dca58de5e0a_calibrate_855042_27.stderr parsl.dataflow.dflow INFO: Standard output for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/4b76bcd0-010c-47f6-ad54-c0caa89597bd_characterizeImage_855042_27.stdout parsl.dataflow.dflow INFO: Standard error for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/4b76bcd0-010c-47f6-ad54-c0caa89597bd_characterizeImage_855042_27.stderr parsl.dataflow.dflow INFO: Task 103 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db10c0304bf013606ba1e7996703d96d with result from task 103 parsl.dataflow.dflow DEBUG: Task 103 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 104 has memoization hash 66f28f51e725d52a76595e8799b8c87d parsl.dataflow.memoization INFO: Task 104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f4e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 104 try 0 launched on executor multi with executor id 802 parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/5cafa3b6-95b5-42da-b58f-4c4afc019fbd_calibrate_855136_3.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/5cafa3b6-95b5-42da-b58f-4c4afc019fbd_calibrate_855136_3.stderr parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/cb9145f4-6ba0-4665-9579-4e7e7d756055_characterizeImage_855136_3.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/cb9145f4-6ba0-4665-9579-4e7e7d756055_characterizeImage_855136_3.stderr parsl.dataflow.dflow INFO: Task 589 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 161fb25842ac90a8260b2ebb90b4d95e with result from task 589 parsl.dataflow.dflow DEBUG: Task 589 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 590 has memoization hash 9816c4a87fb3bce7c9c3033a597d58da parsl.dataflow.memoization INFO: Task 590 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45742d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 590 try 0 launched on executor multi with executor id 803 parsl.dataflow.dflow INFO: Standard output for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/895e59ca-1c93-478a-af24-b564d7d46af6_calibrate_855042_48.stdout parsl.dataflow.dflow INFO: Standard error for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/895e59ca-1c93-478a-af24-b564d7d46af6_calibrate_855042_48.stderr parsl.dataflow.dflow INFO: Standard output for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/52372e7a-1d04-440d-b649-95dac62cc2a7_characterizeImage_855042_48.stdout parsl.dataflow.dflow INFO: Standard error for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/52372e7a-1d04-440d-b649-95dac62cc2a7_characterizeImage_855042_48.stderr parsl.dataflow.dflow INFO: Task 739 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a59e06baa56b12d815b4afdf1b7d2f67 with result from task 739 parsl.dataflow.dflow DEBUG: Task 739 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 740 has memoization hash 551645b7709bf9d161ab85225f326e4a parsl.dataflow.memoization INFO: Task 740 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 740 try 0 launched on executor multi with executor id 804 parsl.dataflow.dflow INFO: Standard output for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/c7477635-d3ba-4b51-ba27-2e538e100f17_calibrate_855136_50.stdout parsl.dataflow.dflow INFO: Standard error for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/c7477635-d3ba-4b51-ba27-2e538e100f17_calibrate_855136_50.stderr parsl.dataflow.dflow INFO: Standard output for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/bdc0f468-6576-4708-8241-f07f3a9c4ccf_characterizeImage_855136_50.stdout parsl.dataflow.dflow INFO: Standard error for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/bdc0f468-6576-4708-8241-f07f3a9c4ccf_characterizeImage_855136_50.stderr parsl.dataflow.dflow INFO: Task 112 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 420ae5c45705c55896b50a68ae15d0ca with result from task 112 parsl.dataflow.dflow DEBUG: Task 112 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 113 has memoization hash fb9a9e3613ee0fe04d43e8aa53dcddca parsl.dataflow.memoization INFO: Task 113 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f5760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 113 try 0 launched on executor multi with executor id 805 parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/95bfd1b2-2a95-43f3-ba18-e25b6e3d544d_calibrate_855138_21.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/95bfd1b2-2a95-43f3-ba18-e25b6e3d544d_calibrate_855138_21.stderr parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/2f786d01-1cb7-4db6-a620-94a55caa1e6f_characterizeImage_855138_21.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/2f786d01-1cb7-4db6-a620-94a55caa1e6f_characterizeImage_855138_21.stderr parsl.dataflow.dflow INFO: Task 745 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 051af44c100be5b28bd9f9b9f6a29eca with result from task 745 parsl.dataflow.dflow DEBUG: Task 745 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 746 has memoization hash 169223691d1dbe4e7ab36ce5e2897185 parsl.dataflow.memoization INFO: Task 746 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 746 try 0 launched on executor multi with executor id 806 parsl.dataflow.dflow INFO: Standard output for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/fe932c5e-156a-4c2d-97fc-7485398d23cc_calibrate_855043_15.stdout parsl.dataflow.dflow INFO: Standard error for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/fe932c5e-156a-4c2d-97fc-7485398d23cc_calibrate_855043_15.stderr parsl.dataflow.dflow INFO: Standard output for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/8b3a159d-41d3-4144-94d4-216a47942b23_characterizeImage_855043_15.stdout parsl.dataflow.dflow INFO: Standard error for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/8b3a159d-41d3-4144-94d4-216a47942b23_characterizeImage_855043_15.stderr parsl.dataflow.dflow INFO: Task 13 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bffcae9bb77c0349839be3ec0d67e794 with result from task 13 parsl.dataflow.dflow DEBUG: Task 13 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 14 has memoization hash 520b344952e0a71d1b83f8a5354e51d9 parsl.dataflow.memoization INFO: Task 14 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467b2ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 14 try 0 launched on executor multi with executor id 807 parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/e2d20e37-b84b-4151-ba11-3e35372bb4bd_calibrate_855044_4.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/e2d20e37-b84b-4151-ba11-3e35372bb4bd_calibrate_855044_4.stderr parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/0c42e1c5-eb02-40ea-bd49-1d8f5fa9c908_characterizeImage_855044_4.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/0c42e1c5-eb02-40ea-bd49-1d8f5fa9c908_characterizeImage_855044_4.stderr parsl.dataflow.dflow INFO: Task 109 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23b41625ce0c53d47a16761cd53e864d with result from task 109 parsl.dataflow.dflow DEBUG: Task 109 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 110 has memoization hash 19dd6640b6da2ed60241ebd12f129bdb parsl.dataflow.memoization INFO: Task 110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f5440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 110 try 0 launched on executor multi with executor id 808 parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/bf8a19f7-74d8-44e4-b505-a3c3a379f8a7_calibrate_855044_21.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/bf8a19f7-74d8-44e4-b505-a3c3a379f8a7_calibrate_855044_21.stderr parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/327cb6b1-44ba-440f-a947-e7929b0e2e1c_characterizeImage_855044_21.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/327cb6b1-44ba-440f-a947-e7929b0e2e1c_characterizeImage_855044_21.stderr parsl.dataflow.dflow INFO: Task 415 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 705f2243b677898262b839c1dbdeb047 with result from task 415 parsl.dataflow.dflow DEBUG: Task 415 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 416 has memoization hash 7414d049fbe8eb9e63125ce5dde2a860 parsl.dataflow.memoization INFO: Task 416 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a4400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 416 try 0 launched on executor multi with executor id 809 parsl.dataflow.dflow INFO: Standard output for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/07ffbe8d-af9c-47e1-b701-8bc202ddaed9_calibrate_855042_12.stdout parsl.dataflow.dflow INFO: Standard error for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/07ffbe8d-af9c-47e1-b701-8bc202ddaed9_calibrate_855042_12.stderr parsl.dataflow.dflow INFO: Standard output for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/b2bd430f-a50f-44fc-a842-856b57745a76_characterizeImage_855042_12.stdout parsl.dataflow.dflow INFO: Standard error for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/b2bd430f-a50f-44fc-a842-856b57745a76_characterizeImage_855042_12.stderr parsl.dataflow.dflow INFO: Task 148 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7bac5eb8c93568841d05d9ccb8b921db with result from task 148 parsl.dataflow.dflow DEBUG: Task 148 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 149 has memoization hash 6a80bb716a65a1202d466c5bdf3d20a1 parsl.dataflow.memoization INFO: Task 149 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f7ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 149 try 0 launched on executor multi with executor id 810 parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/775b497a-1bce-4ca5-8d2e-73cfe8735ee8_calibrate_855138_37.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/775b497a-1bce-4ca5-8d2e-73cfe8735ee8_calibrate_855138_37.stderr parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/ee82dcd2-77a5-4432-8f0e-6f9dfbef9e75_characterizeImage_855138_37.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/ee82dcd2-77a5-4432-8f0e-6f9dfbef9e75_characterizeImage_855138_37.stderr parsl.dataflow.dflow INFO: Task 790 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c10b95635a11785a695ebe4b8b8e88d with result from task 790 parsl.dataflow.dflow DEBUG: Task 790 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 791 has memoization hash 9a331f2827c1e2de40c250019f2f1b90 parsl.dataflow.memoization INFO: Task 791 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 791 try 0 launched on executor multi with executor id 811 parsl.dataflow.dflow INFO: Standard output for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/daed5e80-2a0b-4e5d-a5ef-8972ba41dd3a_calibrate_855044_35.stdout parsl.dataflow.dflow INFO: Standard error for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/daed5e80-2a0b-4e5d-a5ef-8972ba41dd3a_calibrate_855044_35.stderr parsl.dataflow.dflow INFO: Standard output for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/fb4cfe25-d82e-49df-a6f9-207be186bc25_characterizeImage_855044_35.stdout parsl.dataflow.dflow INFO: Standard error for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/fb4cfe25-d82e-49df-a6f9-207be186bc25_characterizeImage_855044_35.stderr parsl.dataflow.dflow INFO: Task 94 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1e7393065d64717fc20b42df238730f with result from task 94 parsl.dataflow.dflow DEBUG: Task 94 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 95 has memoization hash a18a11de59bdc7edecab58dd665c45ca parsl.dataflow.memoization INFO: Task 95 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f44a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 95 try 0 launched on executor multi with executor id 812 parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/0bf9594c-4601-4c04-bef1-825fe56383d4_calibrate_855137_30.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/0bf9594c-4601-4c04-bef1-825fe56383d4_calibrate_855137_30.stderr parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/ceb0c9a2-7712-48b9-9626-5acce65e4ad1_characterizeImage_855137_30.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/ceb0c9a2-7712-48b9-9626-5acce65e4ad1_characterizeImage_855137_30.stderr parsl.dataflow.dflow INFO: Task 430 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1be0d67a211de1f24d4a8973f357eaab with result from task 430 parsl.dataflow.dflow DEBUG: Task 430 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 431 has memoization hash 65d62ecc7e404add615ea379e142920a parsl.dataflow.memoization INFO: Task 431 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a5260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 431 try 0 launched on executor multi with executor id 813 parsl.dataflow.dflow INFO: Standard output for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/701f1c83-c2f9-45ec-95d7-98477fca9cf8_calibrate_855043_4.stdout parsl.dataflow.dflow INFO: Standard error for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/701f1c83-c2f9-45ec-95d7-98477fca9cf8_calibrate_855043_4.stderr parsl.dataflow.dflow INFO: Standard output for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/b3cec1e7-ef40-4f00-b70b-125a3614f4c8_characterizeImage_855043_4.stdout parsl.dataflow.dflow INFO: Standard error for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/b3cec1e7-ef40-4f00-b70b-125a3614f4c8_characterizeImage_855043_4.stderr parsl.dataflow.dflow INFO: Task 412 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79230e1a5ec12d5c630eefa3c05daab9 with result from task 412 parsl.dataflow.dflow DEBUG: Task 412 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 413 has memoization hash b1f6addcec4819fd9bc307768b931835 parsl.dataflow.memoization INFO: Task 413 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a7060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 413 try 0 launched on executor multi with executor id 814 parsl.dataflow.dflow INFO: Standard output for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/b273e99e-6cc4-4147-8538-e81cf13615db_calibrate_855136_38.stdout parsl.dataflow.dflow INFO: Standard error for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/b273e99e-6cc4-4147-8538-e81cf13615db_calibrate_855136_38.stderr parsl.dataflow.dflow INFO: Standard output for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/693466bf-af0d-4d00-904d-d5d302fc91e1_characterizeImage_855136_38.stdout parsl.dataflow.dflow INFO: Standard error for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/693466bf-af0d-4d00-904d-d5d302fc91e1_characterizeImage_855136_38.stderr parsl.dataflow.dflow INFO: Task 121 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91b6558a4f587b9c8f55b9e34035b217 with result from task 121 parsl.dataflow.dflow DEBUG: Task 121 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 122 has memoization hash ded91c991ac29cb585d792e5f92faf49 parsl.dataflow.memoization INFO: Task 122 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f60c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 122 try 0 launched on executor multi with executor id 815 parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/7d33d7f9-651e-4e93-a8c3-7efd5e25d2b7_calibrate_855044_57.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/7d33d7f9-651e-4e93-a8c3-7efd5e25d2b7_calibrate_855044_57.stderr parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/d61962d4-bdb2-4fc3-8552-48621f83cd07_characterizeImage_855044_57.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/d61962d4-bdb2-4fc3-8552-48621f83cd07_characterizeImage_855044_57.stderr parsl.dataflow.dflow INFO: Task 805 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 37319c635b66bc57e11a43789de61a7e with result from task 805 parsl.dataflow.dflow DEBUG: Task 805 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 806 has memoization hash 3eabfa1a273b30f42b4412cbafafeb11 parsl.dataflow.memoization INFO: Task 806 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 806 try 0 launched on executor multi with executor id 816 parsl.dataflow.dflow INFO: Standard output for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/fdb40be7-6ec1-4a81-9d30-02ec94e7091e_calibrate_855136_6.stdout parsl.dataflow.dflow INFO: Standard error for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/fdb40be7-6ec1-4a81-9d30-02ec94e7091e_calibrate_855136_6.stderr parsl.dataflow.dflow INFO: Standard output for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/f2b0edde-d5fd-4938-84bd-5906d05715e8_characterizeImage_855136_6.stdout parsl.dataflow.dflow INFO: Standard error for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/f2b0edde-d5fd-4938-84bd-5906d05715e8_characterizeImage_855136_6.stderr parsl.dataflow.dflow INFO: Task 97 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d28ac0e9240d793e07c6d97ffe98b68 with result from task 97 parsl.dataflow.dflow DEBUG: Task 97 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 98 has memoization hash 45e8ed9b1f1296e9655dac000df0cbc5 parsl.dataflow.memoization INFO: Task 98 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f47c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 98 try 0 launched on executor multi with executor id 817 parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/c2018014-5353-4b60-903a-e8153c50efb3_calibrate_855042_55.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/c2018014-5353-4b60-903a-e8153c50efb3_calibrate_855042_55.stderr parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/65524315-6250-4b13-baeb-6b3881e6ffeb_characterizeImage_855042_55.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/65524315-6250-4b13-baeb-6b3881e6ffeb_characterizeImage_855042_55.stderr parsl.dataflow.dflow INFO: Task 133 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14d4850e1684ac9ec708a1393c5b7ff1 with result from task 133 parsl.dataflow.dflow DEBUG: Task 133 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 134 has memoization hash 09228c2ba7519c78995eb088b447c8b7 parsl.dataflow.memoization INFO: Task 134 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f6d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 134 try 0 launched on executor multi with executor id 818 parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/759b5834-92e5-4c07-92cd-e0162e423dff_calibrate_855044_45.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/759b5834-92e5-4c07-92cd-e0162e423dff_calibrate_855044_45.stderr parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/12083d16-f0d7-4066-9e0c-c0e520449e58_characterizeImage_855044_45.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/12083d16-f0d7-4066-9e0c-c0e520449e58_characterizeImage_855044_45.stderr parsl.channels.local.local WARNING: Execution of command 'squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378'' failed due to Command 'squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378'' timed out after 10 seconds parsl.utils ERROR: Callback threw an exception - logging and proceeding anyway Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/utils.py", line 341, in make_callback self.callback(*self.cb_args) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/job_status_poller.py", line 114, in poll self._update_state() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/job_status_poller.py", line 121, in _update_state item.poll(now) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/job_status_poller.py", line 47, in poll self._status = self._executor.status() ^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/executors/status_handling.py", line 109, in status status = self._make_status_dict(block_ids, self._provider.status(job_ids)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/gscratch/dirac/shared/opt/proc_lsst/python/proc_lsst/multi.py", line 190, in status statuses += self.providers[provider].status([job_id]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/providers/cluster_provider.py", line 142, in status self._status() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/providers/slurm/slurm.py", line 157, in _status retcode, stdout, stderr = self.execute_wait(cmd) ^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/providers/cluster_provider.py", line 83, in execute_wait return self.channel.execute_wait(cmd, t) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/channels/local/local.py", line 71, in execute_wait (stdout, stderr) = proc.communicate(timeout=walltime) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/subprocess.py", line 1209, in communicate stdout, stderr = self._communicate(input, endtime, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/subprocess.py", line 2114, in _communicate self._check_timeout(endtime, orig_timeout, stdout, stderr) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/subprocess.py", line 1253, in _check_timeout raise TimeoutExpired( subprocess.TimeoutExpired: Command 'squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378'' timed out after 10 seconds 2024-03-08 07:18:14 proc_lsst.multi:146 [INFO] found job 75775 in provider local proc_lsst.multi INFO: found job 75775 in provider local parsl.dataflow.dflow INFO: Task 829 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e08ee43c6c8927fb9b735390efcd87c with result from task 829 parsl.dataflow.dflow DEBUG: Task 829 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 830 has memoization hash 4838ef3382ade9a5a20fa840212591dd parsl.dataflow.memoization INFO: Task 830 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45664900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 830 try 0 launched on executor multi with executor id 819 parsl.dataflow.dflow INFO: Standard output for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/7e793946-2771-460f-8273-72aeb250275d_calibrate_855136_5.stdout parsl.dataflow.dflow INFO: Standard error for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/7e793946-2771-460f-8273-72aeb250275d_calibrate_855136_5.stderr parsl.dataflow.dflow INFO: Standard output for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/b42b9a6b-38cd-4e24-94df-ff8e2a81559b_characterizeImage_855136_5.stdout parsl.dataflow.dflow INFO: Standard error for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/b42b9a6b-38cd-4e24-94df-ff8e2a81559b_characterizeImage_855136_5.stderr parsl.dataflow.dflow INFO: Task 334 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 60285212fd19a3064d69c4710514c091 with result from task 334 parsl.dataflow.dflow DEBUG: Task 334 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 335 has memoization hash c2965993b3cf1a4d6bcb19ea48031737 parsl.dataflow.memoization INFO: Task 335 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 335 try 0 launched on executor multi with executor id 820 parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/f3af6665-f039-4ca8-b874-eb372b9ba647_calibrate_855138_24.stdout parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/f3af6665-f039-4ca8-b874-eb372b9ba647_calibrate_855138_24.stderr parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/0460f431-375c-473f-b5e5-68d57a661e4e_characterizeImage_855138_24.stdout parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/0460f431-375c-473f-b5e5-68d57a661e4e_characterizeImage_855138_24.stderr 2024-03-08 07:18:14 proc_lsst.multi:146 [INFO] found job 16912283 in provider astro proc_lsst.multi INFO: found job 16912283 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 982 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ddb82f0f133712abf8410789907a113 with result from task 982 parsl.dataflow.dflow DEBUG: Task 982 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 983 has memoization hash 40b70bcec417858a96b30b3573ea8347 parsl.dataflow.memoization INFO: Task 983 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454f9800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 983 try 0 launched on executor multi with executor id 821 parsl.dataflow.dflow INFO: Standard output for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/93b0e79b-3cad-4152-856e-1ac830cc0731_calibrate_855138_31.stdout parsl.dataflow.dflow INFO: Standard error for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/93b0e79b-3cad-4152-856e-1ac830cc0731_calibrate_855138_31.stderr parsl.dataflow.dflow INFO: Standard output for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/9d644d4e-ecb9-4d8c-b1dc-3aabaf34b0d7_characterizeImage_855138_31.stdout parsl.dataflow.dflow INFO: Standard error for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/9d644d4e-ecb9-4d8c-b1dc-3aabaf34b0d7_characterizeImage_855138_31.stderr parsl.dataflow.dflow INFO: Task 898 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c79108b3deb4faf1e4ecf97d02a7645 with result from task 898 parsl.dataflow.dflow DEBUG: Task 898 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 899 has memoization hash 555a0eabc9ecdcbe33892af8f174c107 parsl.dataflow.memoization INFO: Task 899 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454989a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 899 try 0 launched on executor multi with executor id 822 parsl.dataflow.dflow INFO: Standard output for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/8af92a7f-f077-460b-80fb-4646cddd3c0e_calibrate_855136_52.stdout parsl.dataflow.dflow INFO: Standard error for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/8af92a7f-f077-460b-80fb-4646cddd3c0e_calibrate_855136_52.stderr parsl.dataflow.dflow INFO: Standard output for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/49336d7b-2f2f-40f1-a61d-f311e9602a96_characterizeImage_855136_52.stdout parsl.dataflow.dflow INFO: Standard error for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/49336d7b-2f2f-40f1-a61d-f311e9602a96_characterizeImage_855136_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:18:15 proc_lsst.multi:146 [INFO] found job 16912284 in provider astro proc_lsst.multi INFO: found job 16912284 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 607 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a8a73f385c7c808b7183d1ac78cb328 with result from task 607 parsl.dataflow.dflow DEBUG: Task 607 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 608 has memoization hash 55033658773cefd60667d29cd7c7f7e5 parsl.dataflow.memoization INFO: Task 608 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45741120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 608 try 0 launched on executor multi with executor id 823 parsl.dataflow.dflow INFO: Standard output for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/6667b1f9-161f-49b4-aed2-dc73d7abe15d_calibrate_855138_5.stdout parsl.dataflow.dflow INFO: Standard error for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/6667b1f9-161f-49b4-aed2-dc73d7abe15d_calibrate_855138_5.stderr parsl.dataflow.dflow INFO: Standard output for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/7ecebbf8-697e-4c14-acfe-05eb3a13e303_characterizeImage_855138_5.stdout parsl.dataflow.dflow INFO: Standard error for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/7ecebbf8-697e-4c14-acfe-05eb3a13e303_characterizeImage_855138_5.stderr parsl.dataflow.dflow INFO: Task 238 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b720fbe683304386313306d3821ec2b with result from task 238 parsl.dataflow.dflow DEBUG: Task 238 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 239 has memoization hash 738a8e569e15ef67737e6e019143c5a6 parsl.dataflow.memoization INFO: Task 239 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46835b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 239 try 0 launched on executor multi with executor id 824 parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/9f1494fd-6b37-459a-a5f7-061654f8b538_calibrate_855138_4.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/9f1494fd-6b37-459a-a5f7-061654f8b538_calibrate_855138_4.stderr parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/73939ba5-429c-4b67-acb6-3db2c8aabb35_characterizeImage_855138_4.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/73939ba5-429c-4b67-acb6-3db2c8aabb35_characterizeImage_855138_4.stderr parsl.dataflow.dflow INFO: Task 736 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29faed971da8e617fc962788950e23d5 with result from task 736 parsl.dataflow.dflow DEBUG: Task 736 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 737 has memoization hash 0e46a3e92ce7c98cd77604cd77a39366 parsl.dataflow.memoization INFO: Task 737 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 737 try 0 launched on executor multi with executor id 825 parsl.dataflow.dflow INFO: Standard output for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/3f641e5d-4f9b-48b3-8a92-14b66091cea6_calibrate_855044_6.stdout parsl.dataflow.dflow INFO: Standard error for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/3f641e5d-4f9b-48b3-8a92-14b66091cea6_calibrate_855044_6.stderr parsl.dataflow.dflow INFO: Standard output for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/360351ff-fc5d-4d3b-85f7-440d48d8eafd_characterizeImage_855044_6.stdout parsl.dataflow.dflow INFO: Standard error for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/360351ff-fc5d-4d3b-85f7-440d48d8eafd_characterizeImage_855044_6.stderr parsl.dataflow.dflow INFO: Task 613 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 19cc8ca0c6556a39ee1a7a8d073b6be8 with result from task 613 parsl.dataflow.dflow DEBUG: Task 613 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 614 has memoization hash d3624f130a3d64463ec9f428def640b9 parsl.dataflow.memoization INFO: Task 614 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa457418a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 614 try 0 launched on executor multi with executor id 826 parsl.dataflow.dflow INFO: Standard output for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/d25359d1-eff0-4228-b21c-e8b78e55179d_calibrate_855044_29.stdout parsl.dataflow.dflow INFO: Standard error for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/d25359d1-eff0-4228-b21c-e8b78e55179d_calibrate_855044_29.stderr parsl.dataflow.dflow INFO: Standard output for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/bdd7c8a3-e23f-41f3-a619-b4b50785512f_characterizeImage_855044_29.stdout parsl.dataflow.dflow INFO: Standard error for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/bdd7c8a3-e23f-41f3-a619-b4b50785512f_characterizeImage_855044_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:18:16 proc_lsst.multi:146 [INFO] found job 16912285 in provider astro proc_lsst.multi INFO: found job 16912285 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 868 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 655a13d0b7b1fd8de399881421a13d19 with result from task 868 parsl.dataflow.dflow DEBUG: Task 868 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 869 has memoization hash 7fbc56499d88e1f3be308c2ef9607306 parsl.dataflow.memoization INFO: Task 869 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45666840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 869 try 0 launched on executor multi with executor id 827 parsl.dataflow.dflow INFO: Standard output for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/ebd9b366-51d3-4259-a2a7-68d709e50357_calibrate_855044_13.stdout parsl.dataflow.dflow INFO: Standard error for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/ebd9b366-51d3-4259-a2a7-68d709e50357_calibrate_855044_13.stderr parsl.dataflow.dflow INFO: Standard output for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/c08d2d6d-b768-4316-a6e8-a08f7d04471d_characterizeImage_855044_13.stdout parsl.dataflow.dflow INFO: Standard error for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/c08d2d6d-b768-4316-a6e8-a08f7d04471d_characterizeImage_855044_13.stderr parsl.dataflow.dflow INFO: Task 139 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d316f8d6bab4ae779e2f370de47c9277 with result from task 139 parsl.dataflow.dflow DEBUG: Task 139 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 140 has memoization hash c9bb4a8152549d827e8829fa08061e6c parsl.dataflow.memoization INFO: Task 140 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f7380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 140 try 0 launched on executor multi with executor id 828 parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/4bc2e91d-3d07-45e8-8a78-a414bb2ce606_calibrate_855044_34.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/4bc2e91d-3d07-45e8-8a78-a414bb2ce606_calibrate_855044_34.stderr parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/69f0e87f-a264-4efb-9dbe-c7dd4f51af85_characterizeImage_855044_34.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/69f0e87f-a264-4efb-9dbe-c7dd4f51af85_characterizeImage_855044_34.stderr parsl.dataflow.dflow INFO: Task 157 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed9c59e73e3d9a3b855e260b52b2186f with result from task 157 parsl.dataflow.dflow DEBUG: Task 157 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 158 has memoization hash 76f38a5b15e95d95f2493a52c3b51854 parsl.dataflow.memoization INFO: Task 158 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46814680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 158 try 0 launched on executor multi with executor id 829 parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/ca87c467-32bc-45db-9a62-6fdc570d71ff_calibrate_855043_55.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/ca87c467-32bc-45db-9a62-6fdc570d71ff_calibrate_855043_55.stderr parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/ab985c4a-d0a3-42a2-b0f0-91438c6d5169_characterizeImage_855043_55.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/ab985c4a-d0a3-42a2-b0f0-91438c6d5169_characterizeImage_855043_55.stderr parsl.dataflow.dflow INFO: Task 815 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f68922ca7ecf948b5cb4f5bd01b5dca1 with result from task 815 parsl.dataflow.dflow DEBUG: Task 815 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/f6aa7f37-4add-4cd8-8d1d-eeac59446ae9_calibrate_855043_31.stdout parsl.dataflow.dflow INFO: Standard error for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/f6aa7f37-4add-4cd8-8d1d-eeac59446ae9_calibrate_855043_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:18:17 proc_lsst.multi:146 [INFO] found job 16912286 in provider astro proc_lsst.multi INFO: found job 16912286 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 124 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9302988dbe05646ef3e4ba6d6f251b65 with result from task 124 parsl.dataflow.dflow DEBUG: Task 124 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 125 has memoization hash 6b4fb1b0c14b0d264612348235235541 parsl.dataflow.memoization INFO: Task 125 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f63e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 125 try 0 launched on executor multi with executor id 830 parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/ea18e29b-cd67-4cc7-b9fe-3c4a5fd6b746_calibrate_855137_51.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/ea18e29b-cd67-4cc7-b9fe-3c4a5fd6b746_calibrate_855137_51.stderr parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/9aa30944-e30a-4061-b9f3-a5d138b64ece_characterizeImage_855137_51.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/9aa30944-e30a-4061-b9f3-a5d138b64ece_characterizeImage_855137_51.stderr parsl.dataflow.dflow INFO: Task 181 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2e797a59871f6d34372f3b3745a98fc with result from task 181 parsl.dataflow.dflow DEBUG: Task 181 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 182 has memoization hash f33689a8dfcb01649df9847486e6c18b parsl.dataflow.memoization INFO: Task 182 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46815f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 182 try 0 launched on executor multi with executor id 831 parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/4a8f38ee-ed1d-40d7-9ce2-c81ee7faae3b_calibrate_855044_58.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/4a8f38ee-ed1d-40d7-9ce2-c81ee7faae3b_calibrate_855044_58.stderr parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/66943aca-617f-482c-be6a-bda81775548a_characterizeImage_855044_58.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/66943aca-617f-482c-be6a-bda81775548a_characterizeImage_855044_58.stderr parsl.dataflow.dflow INFO: Task 778 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b17b9396c5aa649a0ce26487b58ac56 with result from task 778 parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:18:18 proc_lsst.multi:146 [INFO] found job 16912287 in provider astro proc_lsst.multi INFO: found job 16912287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow DEBUG: Task 778 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 779 has memoization hash 20ac51adba9114c515b77b4469572ef6 parsl.dataflow.memoization INFO: Task 779 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 779 try 0 launched on executor multi with executor id 832 parsl.dataflow.dflow INFO: Standard output for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/64e102e7-38f2-4033-b50b-4ee1cb0c546c_calibrate_855044_26.stdout parsl.dataflow.dflow INFO: Standard error for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/64e102e7-38f2-4033-b50b-4ee1cb0c546c_calibrate_855044_26.stderr parsl.dataflow.dflow INFO: Standard output for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/7839a0a9-b0da-4e37-b8f9-eba6dfac75de_characterizeImage_855044_26.stdout parsl.dataflow.dflow INFO: Standard error for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/7839a0a9-b0da-4e37-b8f9-eba6dfac75de_characterizeImage_855044_26.stderr parsl.dataflow.dflow INFO: Task 388 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7683ee44c9d945153bcae6861bbea73a with result from task 388 parsl.dataflow.dflow DEBUG: Task 388 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 389 has memoization hash d5b8d45db34ae0d81c36eff3cb20bcf8 parsl.dataflow.memoization INFO: Task 389 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4687b6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 389 try 0 launched on executor multi with executor id 833 parsl.dataflow.dflow INFO: Standard output for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/d28fdede-3ca2-4fb8-ae66-9c64d5f82276_calibrate_855042_51.stdout parsl.dataflow.dflow INFO: Standard error for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/d28fdede-3ca2-4fb8-ae66-9c64d5f82276_calibrate_855042_51.stderr parsl.dataflow.dflow INFO: Standard output for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/408785e0-e2c6-45d7-bbbb-9bd43d4d2f86_characterizeImage_855042_51.stdout parsl.dataflow.dflow INFO: Standard error for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/408785e0-e2c6-45d7-bbbb-9bd43d4d2f86_characterizeImage_855042_51.stderr parsl.dataflow.dflow INFO: Task 877 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d89f65c149ec66da160bc7b73ccfe6ef with result from task 877 parsl.dataflow.dflow DEBUG: Task 877 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 878 has memoization hash e948897629e422307511c918b926eea3 parsl.dataflow.memoization INFO: Task 878 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45667240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 878 try 0 launched on executor multi with executor id 834 parsl.dataflow.dflow INFO: Standard output for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/89412d7e-1980-42ce-903d-24097dfcb2ed_calibrate_855137_40.stdout parsl.dataflow.dflow INFO: Standard error for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/89412d7e-1980-42ce-903d-24097dfcb2ed_calibrate_855137_40.stderr parsl.dataflow.dflow INFO: Standard output for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/a6bee093-3a37-4f81-b268-886aa0d97658_characterizeImage_855137_40.stdout parsl.dataflow.dflow INFO: Standard error for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/a6bee093-3a37-4f81-b268-886aa0d97658_characterizeImage_855137_40.stderr parsl.dataflow.dflow INFO: Task 232 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a83f08ae137ede0cb48418596a95e3c6 with result from task 232 parsl.dataflow.dflow DEBUG: Task 232 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 233 has memoization hash 5322c1df78d38b4c73830c9a3ba72e64 parsl.dataflow.memoization INFO: Task 233 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa468354e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 233 try 0 launched on executor multi with executor id 835 parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/0e5cf54e-cc25-49b7-9538-d3077983f08c_calibrate_855044_48.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/0e5cf54e-cc25-49b7-9538-d3077983f08c_calibrate_855044_48.stderr parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/d8eec138-010f-4e97-9244-64a576585e53_characterizeImage_855044_48.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/d8eec138-010f-4e97-9244-64a576585e53_characterizeImage_855044_48.stderr parsl.dataflow.dflow INFO: Task 73 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f725b75d689a37f3a1e7e86a3d0b1374 with result from task 73 parsl.dataflow.dflow DEBUG: Task 73 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 74 has memoization hash c3aadf7902866e0c66578685c90c88bd parsl.dataflow.memoization INFO: Task 74 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d2e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 74 try 0 launched on executor multi with executor id 836 parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/6f1e82dd-ea9c-4e17-8ca0-fc8d408f2e45_calibrate_855136_13.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/6f1e82dd-ea9c-4e17-8ca0-fc8d408f2e45_calibrate_855136_13.stderr parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/7d77561c-108a-4042-90f6-a7f2c8f923bb_characterizeImage_855136_13.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/7d77561c-108a-4042-90f6-a7f2c8f923bb_characterizeImage_855136_13.stderr parsl.dataflow.dflow INFO: Task 67 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13f1cb49a919c3c87b6c7bff3d2ca7c0 with result from task 67 parsl.dataflow.dflow DEBUG: Task 67 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 68 has memoization hash 63ac3e0c3f30e659a98f949f441a2e03 parsl.dataflow.memoization INFO: Task 68 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d2840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 68 try 0 launched on executor multi with executor id 837 parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/732d9894-9437-4b01-8483-feca2e0a6e9a_calibrate_855136_34.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/732d9894-9437-4b01-8483-feca2e0a6e9a_calibrate_855136_34.stderr parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/10c17075-adc0-44f6-b018-0f25128bc1ad_characterizeImage_855136_34.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/10c17075-adc0-44f6-b018-0f25128bc1ad_characterizeImage_855136_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:18:19 proc_lsst.multi:146 [INFO] found job 16912288 in provider astro proc_lsst.multi INFO: found job 16912288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 835 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 15959937fdda8819794a52c9f646a9ec with result from task 835 parsl.dataflow.dflow DEBUG: Task 835 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 836 has memoization hash efb623ddabc49bd46677215eb10c94a0 parsl.dataflow.memoization INFO: Task 836 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45664f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 836 try 0 launched on executor multi with executor id 838 parsl.dataflow.dflow INFO: Standard output for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/6163dcab-8efd-4efa-a544-44adbbe7fb6a_calibrate_855044_42.stdout parsl.dataflow.dflow INFO: Standard error for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/6163dcab-8efd-4efa-a544-44adbbe7fb6a_calibrate_855044_42.stderr parsl.dataflow.dflow INFO: Standard output for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/e4414610-4f04-4805-bbfd-414f1bdd8183_characterizeImage_855044_42.stdout parsl.dataflow.dflow INFO: Standard error for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/e4414610-4f04-4805-bbfd-414f1bdd8183_characterizeImage_855044_42.stderr parsl.dataflow.dflow INFO: Task 469 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c04cd4e11cde2d117d75cf3281e76a9 with result from task 469 parsl.dataflow.dflow DEBUG: Task 469 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 470 has memoization hash 2ab7a82ebf9d8499547c432fce98a8b6 parsl.dataflow.memoization INFO: Task 470 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456cf380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 470 try 0 launched on executor multi with executor id 839 parsl.dataflow.dflow INFO: Standard output for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/76cd631a-4524-488f-9532-57081b1f0cfe_calibrate_855044_15.stdout parsl.dataflow.dflow INFO: Standard error for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/76cd631a-4524-488f-9532-57081b1f0cfe_calibrate_855044_15.stderr parsl.dataflow.dflow INFO: Standard output for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/04eff46a-0f1f-4f66-a5c3-6403143835cd_characterizeImage_855044_15.stdout parsl.dataflow.dflow INFO: Standard error for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/04eff46a-0f1f-4f66-a5c3-6403143835cd_characterizeImage_855044_15.stderr parsl.dataflow.dflow INFO: Task 754 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e5c7152c8795042dcb221a1c8f08cc9 with result from task 754 parsl.dataflow.dflow DEBUG: Task 754 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 755 has memoization hash f573a3a4ccbf03a8a17e6d1f1fda38fe parsl.dataflow.memoization INFO: Task 755 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d27a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 755 try 0 launched on executor multi with executor id 840 parsl.dataflow.dflow INFO: Standard output for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/5a535596-b295-4a95-9f09-7143dd685eea_calibrate_855042_47.stdout parsl.dataflow.dflow INFO: Standard error for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/5a535596-b295-4a95-9f09-7143dd685eea_calibrate_855042_47.stderr parsl.dataflow.dflow INFO: Standard output for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/aa03a199-289f-476e-900a-c9824b086ad5_characterizeImage_855042_47.stdout parsl.dataflow.dflow INFO: Standard error for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/aa03a199-289f-476e-900a-c9824b086ad5_characterizeImage_855042_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:18:20 proc_lsst.multi:146 [INFO] found job 16912289 in provider astro proc_lsst.multi INFO: found job 16912289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 706 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e11d76bd79d978990813f5a66721df0a with result from task 706 parsl.dataflow.dflow DEBUG: Task 706 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 707 has memoization hash ae31529b58d725e0a566abc6bf6ecf7e parsl.dataflow.memoization INFO: Task 707 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d0860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 707 try 0 launched on executor multi with executor id 841 parsl.dataflow.dflow INFO: Standard output for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/f06d5736-5ca9-4c5d-878d-baea609a23dc_calibrate_855138_41.stdout parsl.dataflow.dflow INFO: Standard error for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/f06d5736-5ca9-4c5d-878d-baea609a23dc_calibrate_855138_41.stderr parsl.dataflow.dflow INFO: Standard output for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/c36d0084-afcb-4ef7-a7e1-40bcbdb9d152_characterizeImage_855138_41.stdout parsl.dataflow.dflow INFO: Standard error for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/c36d0084-afcb-4ef7-a7e1-40bcbdb9d152_characterizeImage_855138_41.stderr parsl.dataflow.dflow INFO: Task 4 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80c50c9bc877bfd12dd664ee4cd46996 with result from task 4 parsl.dataflow.dflow DEBUG: Task 4 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5 has memoization hash e5d41956feb51c0235da461af9ec39d0 parsl.dataflow.memoization INFO: Task 5 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467b2480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5 try 0 launched on executor multi with executor id 842 parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/1e2af3ee-d90c-4e00-b5be-54f7461249b0_calibrate_855137_13.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/1e2af3ee-d90c-4e00-b5be-54f7461249b0_calibrate_855137_13.stderr parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/c88d74be-60a3-45bf-9997-ab1d22296af1_characterizeImage_855137_13.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/c88d74be-60a3-45bf-9997-ab1d22296af1_characterizeImage_855137_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:18:20 proc_lsst.multi:146 [INFO] found job 16912290 in provider astro proc_lsst.multi INFO: found job 16912290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 859 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5fdd3b444b10e4fef8c89586befadf8 with result from task 859 parsl.dataflow.dflow DEBUG: Task 859 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 860 has memoization hash 2584a833cb49451154e5ea2dad3ad723 parsl.dataflow.memoization INFO: Task 860 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45499120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 860 try 0 launched on executor multi with executor id 843 parsl.dataflow.dflow INFO: Standard output for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/bcaf17dc-c1bb-419b-86be-53a5693a78c1_calibrate_855136_32.stdout parsl.dataflow.dflow INFO: Standard error for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/bcaf17dc-c1bb-419b-86be-53a5693a78c1_calibrate_855136_32.stderr parsl.dataflow.dflow INFO: Standard output for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/2d8f5ec9-35d6-4b0a-9e7d-b4575868a39e_characterizeImage_855136_32.stdout parsl.dataflow.dflow INFO: Standard error for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/2d8f5ec9-35d6-4b0a-9e7d-b4575868a39e_characterizeImage_855136_32.stderr parsl.dataflow.dflow INFO: Task 292 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f0a2a10aca1ec98a24f02e34c5e3ff7 with result from task 292 parsl.dataflow.dflow DEBUG: Task 292 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 293 has memoization hash 44df8a7460caf03a6b967ae33bb9f18b parsl.dataflow.memoization INFO: Task 293 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 293 try 0 launched on executor multi with executor id 844 parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/fb852705-b881-4a34-8873-308beefcd0e0_calibrate_855044_12.stdout parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/fb852705-b881-4a34-8873-308beefcd0e0_calibrate_855044_12.stderr parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/9cd770ed-83d4-441d-b93c-c75c12ab3386_characterizeImage_855044_12.stdout parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/9cd770ed-83d4-441d-b93c-c75c12ab3386_characterizeImage_855044_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:18:21 proc_lsst.multi:146 [INFO] found job 16912291 in provider astro proc_lsst.multi INFO: found job 16912291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 65 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 566f3ddb68117498bf98105a3cd410d6 with result from task 65 parsl.dataflow.dflow DEBUG: Task 65 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/fc77bc70-3f39-4916-8f9f-1dac6f13929d_calibrate_855137_31.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/fc77bc70-3f39-4916-8f9f-1dac6f13929d_calibrate_855137_31.stderr parsl.dataflow.dflow INFO: Task 28 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d88c35be37c6f191d0a09d9df87062e with result from task 28 parsl.dataflow.dflow DEBUG: Task 28 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 29 has memoization hash fa3920af2d43c023ca412b830fd4127b parsl.dataflow.memoization INFO: Task 29 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467b3920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 29 try 0 launched on executor multi with executor id 845 parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/b9cffdc3-2530-49eb-97ec-41dea25fb017_calibrate_855137_19.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/b9cffdc3-2530-49eb-97ec-41dea25fb017_calibrate_855137_19.stderr parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/b2536c1b-d8be-4dce-a22e-3b2aa66a65b0_characterizeImage_855137_19.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/b2536c1b-d8be-4dce-a22e-3b2aa66a65b0_characterizeImage_855137_19.stderr parsl.dataflow.dflow INFO: Task 52 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00b363912a81b2f885cc9017b26ceb45 with result from task 52 parsl.dataflow.dflow DEBUG: Task 52 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 53 has memoization hash 5afdb029cd75ed55e21eb7439d8e7a44 parsl.dataflow.memoization INFO: Task 53 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d18a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 53 try 0 launched on executor multi with executor id 846 parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/11315694-7bcf-4a5e-87b5-c6d83320318a_calibrate_855137_34.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/11315694-7bcf-4a5e-87b5-c6d83320318a_calibrate_855137_34.stderr parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/ef798390-5f38-4196-826a-0e634d07ecf7_characterizeImage_855137_34.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/ef798390-5f38-4196-826a-0e634d07ecf7_characterizeImage_855137_34.stderr parsl.dataflow.dflow INFO: Task 205 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a43b8dc195376d9c5ede773b5acbbb2 with result from task 205 parsl.dataflow.dflow DEBUG: Task 205 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 206 has memoization hash 5ffb6fc8aa75d88dc43055b93f37098b parsl.dataflow.memoization INFO: Task 206 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46817880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 206 try 0 launched on executor multi with executor id 847 parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/ea3d9cf7-0428-4761-a66f-c3c19a67e871_calibrate_855044_33.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/ea3d9cf7-0428-4761-a66f-c3c19a67e871_calibrate_855044_33.stderr parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/e26f148d-c591-49ca-9a5c-5cf84a33b4c1_characterizeImage_855044_33.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/e26f148d-c591-49ca-9a5c-5cf84a33b4c1_characterizeImage_855044_33.stderr parsl.dataflow.dflow INFO: Task 482 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e4b2b63d7af4bae6ca99e2502835e5d with result from task 482 parsl.dataflow.dflow DEBUG: Task 482 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/29135289-688d-44d2-9921-b773d886d0b5_calibrate_855044_31.stdout parsl.dataflow.dflow INFO: Standard error for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/29135289-688d-44d2-9921-b773d886d0b5_calibrate_855044_31.stderr parsl.dataflow.dflow INFO: Task 115 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6024ea3c4233b610985126d1a156686c with result from task 115 parsl.dataflow.dflow DEBUG: Task 115 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 116 has memoization hash 5a61e7d9e082307a37210e0aebfff03b parsl.dataflow.memoization INFO: Task 116 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f5a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 116 try 0 launched on executor multi with executor id 848 parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/c333f88c-43bb-4453-91e1-4a3e6a222aa4_calibrate_855136_37.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/c333f88c-43bb-4453-91e1-4a3e6a222aa4_calibrate_855136_37.stderr parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/a3a78a31-5494-49f9-84e7-1ce39cdd32e4_characterizeImage_855136_37.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/a3a78a31-5494-49f9-84e7-1ce39cdd32e4_characterizeImage_855136_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:18:22 proc_lsst.multi:146 [INFO] found job 16912292 in provider astro proc_lsst.multi INFO: found job 16912292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 187 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cbdf5cc0d0b23777c724475dd911ca94 with result from task 187 parsl.dataflow.dflow DEBUG: Task 187 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 188 has memoization hash e636d121812330938bff590699c7b3b7 parsl.dataflow.memoization INFO: Task 188 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa468165c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 188 try 0 launched on executor multi with executor id 849 parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/ef1d4320-4f93-4fbe-b05e-d81ab13cadef_calibrate_855044_44.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/ef1d4320-4f93-4fbe-b05e-d81ab13cadef_calibrate_855044_44.stderr parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/92b997cf-ec4a-4a21-8f3d-aed010c3f96b_characterizeImage_855044_44.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/92b997cf-ec4a-4a21-8f3d-aed010c3f96b_characterizeImage_855044_44.stderr parsl.dataflow.dflow INFO: Task 616 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 773bdedd20116b56fb68dd40679d8489 with result from task 616 parsl.dataflow.dflow DEBUG: Task 616 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 617 has memoization hash 63752b44d97e2d4dce829678c93949e4 parsl.dataflow.memoization INFO: Task 617 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45741b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 617 try 0 launched on executor multi with executor id 850 parsl.dataflow.dflow INFO: Standard output for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/944f0ef1-c6da-4213-b19b-d10b7af6e7ee_calibrate_855136_49.stdout parsl.dataflow.dflow INFO: Standard error for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/944f0ef1-c6da-4213-b19b-d10b7af6e7ee_calibrate_855136_49.stderr parsl.dataflow.dflow INFO: Standard output for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/ea6161f5-ed23-4a67-8519-3b7e412517f7_characterizeImage_855136_49.stdout parsl.dataflow.dflow INFO: Standard error for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/ea6161f5-ed23-4a67-8519-3b7e412517f7_characterizeImage_855136_49.stderr parsl.dataflow.dflow INFO: Task 58 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6f072ca7408d5c7fc82e7e395cd36492 with result from task 58 parsl.dataflow.dflow DEBUG: Task 58 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 59 has memoization hash 9dfcb4a7f31ce1df2970a33cc38cb7bf parsl.dataflow.memoization INFO: Task 59 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d1ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 59 try 0 launched on executor multi with executor id 851 parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/582afeb0-3809-4d7d-bdba-cb1481c6b075_calibrate_855136_62.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/582afeb0-3809-4d7d-bdba-cb1481c6b075_calibrate_855136_62.stderr parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/d6cb7c98-9da1-444d-bacb-75afb618c81a_characterizeImage_855136_62.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/d6cb7c98-9da1-444d-bacb-75afb618c81a_characterizeImage_855136_62.stderr parsl.dataflow.dflow INFO: Task 136 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4359779f50d07b6bb30ddd1a64e8b851 with result from task 136 parsl.dataflow.dflow DEBUG: Task 136 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 137 has memoization hash 20f1e0e3d87e38d724ef51c852d9ce70 parsl.dataflow.memoization INFO: Task 137 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f7060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 137 try 0 launched on executor multi with executor id 852 parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/d9b491ff-4776-40fc-ad2b-8c03e258ca30_calibrate_855137_57.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/d9b491ff-4776-40fc-ad2b-8c03e258ca30_calibrate_855137_57.stderr parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/41023e02-839f-41c9-9219-f53c0366a87b_characterizeImage_855137_57.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/41023e02-839f-41c9-9219-f53c0366a87b_characterizeImage_855137_57.stderr parsl.dataflow.dflow INFO: Task 787 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b15f2c4c8931e897ce7eef958be8dd23 with result from task 787 parsl.dataflow.dflow DEBUG: Task 787 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 788 has memoization hash 0bc96c392a1d308488022913757653c4 parsl.dataflow.memoization INFO: Task 788 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 788 try 0 launched on executor multi with executor id 853 parsl.dataflow.dflow INFO: Standard output for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/2e15d384-6efe-48c7-861b-2c418282916e_calibrate_855137_50.stdout parsl.dataflow.dflow INFO: Standard error for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/2e15d384-6efe-48c7-861b-2c418282916e_calibrate_855137_50.stderr parsl.dataflow.dflow INFO: Standard output for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/55393c2a-c811-47c0-a177-2a393a0b29f6_characterizeImage_855137_50.stdout parsl.dataflow.dflow INFO: Standard error for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/55393c2a-c811-47c0-a177-2a393a0b29f6_characterizeImage_855137_50.stderr parsl.dataflow.dflow INFO: Task 1 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c53c74e0c2de213c82f364722bacb20f with result from task 1 parsl.dataflow.dflow DEBUG: Task 1 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2 has memoization hash 73df80ce85c0976e074cb31fbb1e36a0 parsl.dataflow.memoization INFO: Task 2 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467b2020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2 try 0 launched on executor multi with executor id 854 parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/5f60e8ad-81ab-4855-a2e3-8d87ff625f36_calibrate_855137_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/5f60e8ad-81ab-4855-a2e3-8d87ff625f36_calibrate_855137_15.stderr parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/3a7f69fe-e49e-45fa-8476-ab705543b5a5_characterizeImage_855137_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/3a7f69fe-e49e-45fa-8476-ab705543b5a5_characterizeImage_855137_15.stderr parsl.dataflow.dflow INFO: Task 784 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7dd76d711b92f7b636dabfdc278b205e with result from task 784 parsl.dataflow.dflow DEBUG: Task 784 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 785 has memoization hash ef1fac06942ec68b9755d2bb29715e34 parsl.dataflow.memoization INFO: Task 785 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 785 try 0 launched on executor multi with executor id 855 parsl.dataflow.dflow INFO: Standard output for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/857bf01a-ebe1-4f66-9d8d-6a89b02bccb3_calibrate_855043_57.stdout parsl.dataflow.dflow INFO: Standard error for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/857bf01a-ebe1-4f66-9d8d-6a89b02bccb3_calibrate_855043_57.stderr parsl.dataflow.dflow INFO: Standard output for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/74d32e1e-6641-482d-a536-d1178682d1a8_characterizeImage_855043_57.stdout parsl.dataflow.dflow INFO: Standard error for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/74d32e1e-6641-482d-a536-d1178682d1a8_characterizeImage_855043_57.stderr parsl.dataflow.dflow INFO: Task 49 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d98ed34ad8268e57cc336c70f00f3309 with result from task 49 parsl.dataflow.dflow DEBUG: Task 49 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 50 has memoization hash 7592c2c0ce9e19cc2d5f7d2e92de1b4c parsl.dataflow.memoization INFO: Task 50 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d1580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 50 try 0 launched on executor multi with executor id 856 parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/f3bf14dd-0f16-46f6-bfe4-a02c71f2d141_calibrate_855138_55.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/f3bf14dd-0f16-46f6-bfe4-a02c71f2d141_calibrate_855138_55.stderr parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/e9474207-cb9d-4e42-aa92-bc3afc4336bf_characterizeImage_855138_55.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/e9474207-cb9d-4e42-aa92-bc3afc4336bf_characterizeImage_855138_55.stderr parsl.dataflow.dflow INFO: Task 694 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14bb3193fc8d0388d13bef9c5208908e with result from task 694 parsl.dataflow.dflow DEBUG: Task 694 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 695 has memoization hash 8d800cf713cda3b3ebd5058993029cc6 parsl.dataflow.memoization INFO: Task 695 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 695 try 0 launched on executor multi with executor id 857 parsl.dataflow.dflow INFO: Standard output for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/437ab5fd-1bac-40bd-acc6-ca0ed6d657a3_calibrate_855138_29.stdout parsl.dataflow.dflow INFO: Standard error for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/437ab5fd-1bac-40bd-acc6-ca0ed6d657a3_calibrate_855138_29.stderr parsl.dataflow.dflow INFO: Standard output for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/a29a532a-91c5-4329-b4b6-0f860f45ef50_characterizeImage_855138_29.stdout parsl.dataflow.dflow INFO: Standard error for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/a29a532a-91c5-4329-b4b6-0f860f45ef50_characterizeImage_855138_29.stderr parsl.dataflow.dflow INFO: Task 247 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eefb0091df06eb09ad1cbebbc05a0cfa with result from task 247 parsl.dataflow.dflow DEBUG: Task 247 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 248 has memoization hash 7f5edf0cd4c037d759d72fd476149f71 parsl.dataflow.memoization INFO: Task 248 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46836480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 248 try 0 launched on executor multi with executor id 858 parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/8883a743-dbbb-4d6b-bf1a-a632f3a5854a_calibrate_855136_48.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/8883a743-dbbb-4d6b-bf1a-a632f3a5854a_calibrate_855136_48.stderr parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/3c763949-f455-4191-9c57-e64af0ebdb98_characterizeImage_855136_48.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/3c763949-f455-4191-9c57-e64af0ebdb98_characterizeImage_855136_48.stderr parsl.dataflow.dflow INFO: Task 25 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27de2e521cec85af3a4d51e8f44cf038 with result from task 25 parsl.dataflow.dflow DEBUG: Task 25 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 26 has memoization hash 6b4298426886fc77f48bf7ed862bfbda parsl.dataflow.memoization INFO: Task 26 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467b3ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 26 try 0 launched on executor multi with executor id 859 parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/49200616-d1b0-4a76-a832-e9f87d46f0c4_calibrate_855044_22.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/49200616-d1b0-4a76-a832-e9f87d46f0c4_calibrate_855044_22.stderr parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/b332937c-9ff4-4ee8-b096-8591a8b80ae6_characterizeImage_855044_22.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/b332937c-9ff4-4ee8-b096-8591a8b80ae6_characterizeImage_855044_22.stderr parsl.dataflow.dflow INFO: Task 421 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dfdaed7036a7245c025a626ae79b9dbe with result from task 421 parsl.dataflow.dflow DEBUG: Task 421 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 422 has memoization hash da6f96335b4f9395d27295712b09576b parsl.dataflow.memoization INFO: Task 422 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a4720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 422 try 0 launched on executor multi with executor id 860 parsl.dataflow.dflow INFO: Standard output for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/abc8421d-0aab-44ec-be9c-5b8094e094cf_calibrate_855136_28.stdout parsl.dataflow.dflow INFO: Standard error for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/abc8421d-0aab-44ec-be9c-5b8094e094cf_calibrate_855136_28.stderr parsl.dataflow.dflow INFO: Standard output for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/ca326c19-8625-4753-8a7c-5e60e09448fb_characterizeImage_855136_28.stdout parsl.dataflow.dflow INFO: Standard error for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/ca326c19-8625-4753-8a7c-5e60e09448fb_characterizeImage_855136_28.stderr parsl.dataflow.dflow INFO: Task 832 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b056e220651f1047eb7667ab8af39a9 with result from task 832 parsl.dataflow.dflow DEBUG: Task 832 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 833 has memoization hash b008462d3be20429d4fb8bcb7138b6f4 parsl.dataflow.memoization INFO: Task 833 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45664c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 833 try 0 launched on executor multi with executor id 861 parsl.dataflow.dflow INFO: Standard output for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/8368ef91-bc08-4ea6-852a-b0e95cf63ab2_calibrate_855043_34.stdout parsl.dataflow.dflow INFO: Standard error for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/8368ef91-bc08-4ea6-852a-b0e95cf63ab2_calibrate_855043_34.stderr parsl.dataflow.dflow INFO: Standard output for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/da31d560-b415-4448-a72c-d701f6fc457c_characterizeImage_855043_34.stdout parsl.dataflow.dflow INFO: Standard error for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/da31d560-b415-4448-a72c-d701f6fc457c_characterizeImage_855043_34.stderr parsl.dataflow.dflow INFO: Task 817 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry baad0723413510926538a4a021c5cf9d with result from task 817 parsl.dataflow.dflow DEBUG: Task 817 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 818 has memoization hash 90d7597e9ed2b3ceb6db7410eb1924be parsl.dataflow.memoization INFO: Task 818 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 818 try 0 launched on executor multi with executor id 862 parsl.dataflow.dflow INFO: Standard output for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/93b7738f-b669-4b28-8119-96f0a5348424_calibrate_855043_35.stdout parsl.dataflow.dflow INFO: Standard error for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/93b7738f-b669-4b28-8119-96f0a5348424_calibrate_855043_35.stderr parsl.dataflow.dflow INFO: Standard output for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/ff5738e6-f607-4845-a024-30fee2397171_characterizeImage_855043_35.stdout parsl.dataflow.dflow INFO: Standard error for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/ff5738e6-f607-4845-a024-30fee2397171_characterizeImage_855043_35.stderr parsl.dataflow.dflow INFO: Task 142 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a82a8bd1b7c34ddbdecd6cca772d41e6 with result from task 142 parsl.dataflow.dflow DEBUG: Task 142 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 143 has memoization hash 6409f1f5088a129723b392b9d2c2c19e parsl.dataflow.memoization INFO: Task 143 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f76a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 143 try 0 launched on executor multi with executor id 863 parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/41fe2494-97be-403b-8f2c-119d7d1c31f9_calibrate_855044_24.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/41fe2494-97be-403b-8f2c-119d7d1c31f9_calibrate_855044_24.stderr parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/e9b5c366-bda1-4b4a-9a99-f90ee7af7a3f_characterizeImage_855044_24.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/e9b5c366-bda1-4b4a-9a99-f90ee7af7a3f_characterizeImage_855044_24.stderr parsl.dataflow.dflow INFO: Task 166 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73e28589856c2e71bf4ca91ce258a940 with result from task 166 parsl.dataflow.dflow DEBUG: Task 166 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 167 has memoization hash 1b960b637d560c364e3f13a94477db22 parsl.dataflow.memoization INFO: Task 167 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46814fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 167 try 0 launched on executor multi with executor id 864 parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/9bc30c84-9093-4d8e-b8da-57b7523b8f0e_calibrate_855042_20.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/9bc30c84-9093-4d8e-b8da-57b7523b8f0e_calibrate_855042_20.stderr parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/1def5661-9106-45ed-8663-96a3d09891d5_characterizeImage_855042_20.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/1def5661-9106-45ed-8663-96a3d09891d5_characterizeImage_855042_20.stderr parsl.dataflow.dflow INFO: Task 766 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f285534a51b38b20083bc1c8b1b12fe with result from task 766 parsl.dataflow.dflow DEBUG: Task 766 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 767 has memoization hash cff9678e815c43d1cf9fb3881705b5e6 parsl.dataflow.memoization INFO: Task 767 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d36a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 767 try 0 launched on executor multi with executor id 865 parsl.dataflow.dflow INFO: Standard output for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/1fcea252-194e-4bfb-a3f4-ed61ae09420a_calibrate_855137_44.stdout parsl.dataflow.dflow INFO: Standard error for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/1fcea252-194e-4bfb-a3f4-ed61ae09420a_calibrate_855137_44.stderr parsl.dataflow.dflow INFO: Standard output for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/d4e88c59-8f47-4eae-9c37-a3daa4745c40_characterizeImage_855137_44.stdout parsl.dataflow.dflow INFO: Standard error for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/d4e88c59-8f47-4eae-9c37-a3daa4745c40_characterizeImage_855137_44.stderr parsl.dataflow.dflow INFO: Task 892 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e09ed84ee829845fd8ff196cc050da1 with result from task 892 parsl.dataflow.dflow DEBUG: Task 892 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 893 has memoization hash ce30f26b8da66565d8b9a962c873b28f parsl.dataflow.memoization INFO: Task 893 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45498ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 893 try 0 launched on executor multi with executor id 866 parsl.dataflow.dflow INFO: Standard output for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/6f582040-766c-4073-811a-a423f44e64f7_calibrate_855138_40.stdout parsl.dataflow.dflow INFO: Standard error for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/6f582040-766c-4073-811a-a423f44e64f7_calibrate_855138_40.stderr parsl.dataflow.dflow INFO: Standard output for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/60412ab4-d2b1-4673-9626-0a9697d8aae3_characterizeImage_855138_40.stdout parsl.dataflow.dflow INFO: Standard error for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/60412ab4-d2b1-4673-9626-0a9697d8aae3_characterizeImage_855138_40.stderr parsl.dataflow.dflow INFO: Task 106 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4cb89669799b6056e39b1c0d91523925 with result from task 106 parsl.dataflow.dflow DEBUG: Task 106 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 107 has memoization hash 4bc3ca2c2a209336ccadfb0e3b31044d parsl.dataflow.memoization INFO: Task 107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f5120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 107 try 0 launched on executor multi with executor id 867 parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/c7c4fda7-e1e4-4bf0-87a7-022515dc1cb4_calibrate_855042_14.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/c7c4fda7-e1e4-4bf0-87a7-022515dc1cb4_calibrate_855042_14.stderr parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/52ddafda-9886-4e57-9fc8-7d8b140e300d_characterizeImage_855042_14.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/52ddafda-9886-4e57-9fc8-7d8b140e300d_characterizeImage_855042_14.stderr parsl.dataflow.dflow INFO: Task 314 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7eb4b5b67dafc987760761d0b6187492 with result from task 314 parsl.dataflow.dflow DEBUG: Task 314 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/5ccf1061-84a1-4826-9cc6-b9461d8c7fe9_calibrate_855042_31.stdout parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/5ccf1061-84a1-4826-9cc6-b9461d8c7fe9_calibrate_855042_31.stderr parsl.dataflow.dflow INFO: Task 556 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 58790f9b093f0b26747f8b2591980834 with result from task 556 parsl.dataflow.dflow DEBUG: Task 556 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 557 has memoization hash c31ac87fc83367453aa85371e57f1ef2 parsl.dataflow.memoization INFO: Task 557 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45702160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 557 try 0 launched on executor multi with executor id 868 parsl.dataflow.dflow INFO: Standard output for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/bf6eec03-efc2-4760-90ef-a884f8cf2fb0_calibrate_855042_1.stdout parsl.dataflow.dflow INFO: Standard error for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/bf6eec03-efc2-4760-90ef-a884f8cf2fb0_calibrate_855042_1.stderr parsl.dataflow.dflow INFO: Standard output for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/81ecc660-121d-4c7d-bbbb-ff626d43f903_characterizeImage_855042_1.stdout parsl.dataflow.dflow INFO: Standard error for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/81ecc660-121d-4c7d-bbbb-ff626d43f903_characterizeImage_855042_1.stderr parsl.dataflow.dflow INFO: Task 88 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 90af7b201c7d32fbe0f483efea971dd3 with result from task 88 parsl.dataflow.dflow DEBUG: Task 88 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 89 has memoization hash f3beddf237fc91b5aed6f862f2766589 parsl.dataflow.memoization INFO: Task 89 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d3f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 89 try 0 launched on executor multi with executor id 869 parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/ea463d49-9146-4855-a3ef-eca4ba2f395a_calibrate_855044_28.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/ea463d49-9146-4855-a3ef-eca4ba2f395a_calibrate_855044_28.stderr parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/912929fb-2d8b-4d6e-9832-458a1356f45b_characterizeImage_855044_28.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/912929fb-2d8b-4d6e-9832-458a1356f45b_characterizeImage_855044_28.stderr parsl.dataflow.dflow INFO: Task 76 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5f58bb156b08970cbe5fc644bb6b336 with result from task 76 parsl.dataflow.dflow DEBUG: Task 76 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 77 has memoization hash 5b32b711c445421797c05af8f7f2951d parsl.dataflow.memoization INFO: Task 77 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d31a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 77 try 0 launched on executor multi with executor id 870 parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/8a3a873e-13dc-4b01-a857-42d0ce6a273d_calibrate_855136_44.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/8a3a873e-13dc-4b01-a857-42d0ce6a273d_calibrate_855136_44.stderr parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/1323f941-885e-4fd3-8d78-5f903848d918_characterizeImage_855136_44.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/1323f941-885e-4fd3-8d78-5f903848d918_characterizeImage_855136_44.stderr parsl.dataflow.dflow INFO: Task 160 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4cbf92743d38537927537e928bebe9c4 with result from task 160 parsl.dataflow.dflow DEBUG: Task 160 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 161 has memoization hash 9ace94b5c75f7c6420e728e845fe9183 parsl.dataflow.memoization INFO: Task 161 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa468149a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 161 try 0 launched on executor multi with executor id 871 parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/2a1a9977-1b21-463a-b47e-a43a961a36b8_calibrate_855138_57.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/2a1a9977-1b21-463a-b47e-a43a961a36b8_calibrate_855138_57.stderr parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/4010c5aa-b05f-42bf-8abc-a7bf4e73a658_characterizeImage_855138_57.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/4010c5aa-b05f-42bf-8abc-a7bf4e73a658_characterizeImage_855138_57.stderr parsl.dataflow.dflow INFO: Task 853 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2129bfca05683b0c0517b023c4d390e with result from task 853 parsl.dataflow.dflow DEBUG: Task 853 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 854 has memoization hash 2a3935e2b2004fd0e6fea0f6961bd22c parsl.dataflow.memoization INFO: Task 854 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45666020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 854 try 0 launched on executor multi with executor id 872 parsl.dataflow.dflow INFO: Standard output for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/49dbf0c8-9dbe-4405-85a1-969c39779016_calibrate_855137_16.stdout parsl.dataflow.dflow INFO: Standard error for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/49dbf0c8-9dbe-4405-85a1-969c39779016_calibrate_855137_16.stderr parsl.dataflow.dflow INFO: Standard output for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/edb9261f-9908-4593-b130-ca1dad030cb2_characterizeImage_855137_16.stdout parsl.dataflow.dflow INFO: Standard error for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/edb9261f-9908-4593-b130-ca1dad030cb2_characterizeImage_855137_16.stderr parsl.dataflow.dflow INFO: Task 697 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9459e7f843614f2e3f577e31b619fe1c with result from task 697 parsl.dataflow.dflow DEBUG: Task 697 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 698 has memoization hash 4fd3d52870cf687cc42143240affbd14 parsl.dataflow.memoization INFO: Task 698 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 698 try 0 launched on executor multi with executor id 873 parsl.dataflow.dflow INFO: Standard output for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/52121c8e-ac69-4456-b5be-d5b07bb057e7_calibrate_855137_8.stdout parsl.dataflow.dflow INFO: Standard error for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/52121c8e-ac69-4456-b5be-d5b07bb057e7_calibrate_855137_8.stderr parsl.dataflow.dflow INFO: Standard output for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/f40064a1-c578-4ea6-b34a-ddc66844b434_characterizeImage_855137_8.stdout parsl.dataflow.dflow INFO: Standard error for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/f40064a1-c578-4ea6-b34a-ddc66844b434_characterizeImage_855137_8.stderr parsl.dataflow.dflow INFO: Task 871 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1132960f11df702ece8695ea5929157b with result from task 871 parsl.dataflow.dflow DEBUG: Task 871 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 872 has memoization hash 90ee696a558599d1d83929c352cefc4c parsl.dataflow.memoization INFO: Task 872 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45666b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 872 try 0 launched on executor multi with executor id 874 parsl.dataflow.dflow INFO: Standard output for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/0644fab4-4c76-4339-b03a-6deaaa85e837_calibrate_855043_20.stdout parsl.dataflow.dflow INFO: Standard error for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/0644fab4-4c76-4339-b03a-6deaaa85e837_calibrate_855043_20.stderr parsl.dataflow.dflow INFO: Standard output for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/2f6491fe-d2e3-48d1-b91a-1f4b39a87e2d_characterizeImage_855043_20.stdout parsl.dataflow.dflow INFO: Standard error for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/2f6491fe-d2e3-48d1-b91a-1f4b39a87e2d_characterizeImage_855043_20.stderr parsl.dataflow.dflow INFO: Task 178 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c4e59f2692aacbc2679ea08240abb5ea with result from task 178 parsl.dataflow.dflow DEBUG: Task 178 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 179 has memoization hash d2a7999a1a0e97490b94f4511e24bbfc parsl.dataflow.memoization INFO: Task 179 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46815c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 179 try 0 launched on executor multi with executor id 875 parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/1074de16-0c6b-4d47-ade9-2c9c951b36da_calibrate_855137_62.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/1074de16-0c6b-4d47-ade9-2c9c951b36da_calibrate_855137_62.stderr parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/1e99a0ed-f7c4-4fdc-a136-60284dcc1c0a_characterizeImage_855137_62.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/1e99a0ed-f7c4-4fdc-a136-60284dcc1c0a_characterizeImage_855137_62.stderr parsl.dataflow.dflow INFO: Task 913 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b30b5f5a6e27b6f460e3f7529576354 with result from task 913 parsl.dataflow.dflow DEBUG: Task 913 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 914 has memoization hash 4441f72f3af1cf62f4b6f570965b4a63 parsl.dataflow.memoization INFO: Task 914 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4549a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 914 try 0 launched on executor multi with executor id 876 parsl.dataflow.dflow INFO: Standard output for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/12dbde3a-504a-4689-9af1-4e696ba5deb0_calibrate_855042_24.stdout parsl.dataflow.dflow INFO: Standard error for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/12dbde3a-504a-4689-9af1-4e696ba5deb0_calibrate_855042_24.stderr parsl.dataflow.dflow INFO: Standard output for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/b7a2fe68-a718-4376-b28a-0a829652a19d_characterizeImage_855042_24.stdout parsl.dataflow.dflow INFO: Standard error for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/b7a2fe68-a718-4376-b28a-0a829652a19d_characterizeImage_855042_24.stderr parsl.dataflow.dflow INFO: Task 907 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7767485ceb1823884cfc2dd4fb36c8e0 with result from task 907 parsl.dataflow.dflow DEBUG: Task 907 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 908 has memoization hash f4659a6275f3cb817d88f044188f315b parsl.dataflow.memoization INFO: Task 908 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45499a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 908 try 0 launched on executor multi with executor id 877 parsl.dataflow.dflow INFO: Standard output for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/74c3fb69-5670-478a-a471-2137a9d7dec2_calibrate_855042_8.stdout parsl.dataflow.dflow INFO: Standard error for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/74c3fb69-5670-478a-a471-2137a9d7dec2_calibrate_855042_8.stderr parsl.dataflow.dflow INFO: Standard output for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/c137c492-7af8-47ab-a94c-4743e0c2fa53_characterizeImage_855042_8.stdout parsl.dataflow.dflow INFO: Standard error for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/c137c492-7af8-47ab-a94c-4743e0c2fa53_characterizeImage_855042_8.stderr parsl.dataflow.dflow INFO: Task 448 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73ca16f325173dc87d5774815473ceb9 with result from task 448 parsl.dataflow.dflow DEBUG: Task 448 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 449 has memoization hash ab198a0b102df71c31521c79e004bcbd parsl.dataflow.memoization INFO: Task 449 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a6a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 449 try 0 launched on executor multi with executor id 878 parsl.dataflow.dflow INFO: Standard output for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/1ba9cadd-64dc-4c20-96e4-4f2f8b0d2300_calibrate_855137_49.stdout parsl.dataflow.dflow INFO: Standard error for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/1ba9cadd-64dc-4c20-96e4-4f2f8b0d2300_calibrate_855137_49.stderr parsl.dataflow.dflow INFO: Standard output for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/3dc52591-ef88-4e62-89d4-0bd89989e2fa_characterizeImage_855137_49.stdout parsl.dataflow.dflow INFO: Standard error for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/3dc52591-ef88-4e62-89d4-0bd89989e2fa_characterizeImage_855137_49.stderr parsl.dataflow.dflow INFO: Task 364 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 288b3b06f1ac01c7ffc68a99a0e1bed5 with result from task 364 parsl.dataflow.dflow DEBUG: Task 364 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 365 has memoization hash e94c2e69b7b53c7e9da18d62c443923d parsl.dataflow.memoization INFO: Task 365 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46879ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 365 try 0 launched on executor multi with executor id 879 parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/daa79955-7230-40e2-b1b4-89f0125397fb_calibrate_855043_30.stdout parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/daa79955-7230-40e2-b1b4-89f0125397fb_calibrate_855043_30.stderr parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/e54ec542-eb96-4623-a232-9d4924953c9b_characterizeImage_855043_30.stdout parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/e54ec542-eb96-4623-a232-9d4924953c9b_characterizeImage_855043_30.stderr parsl.dataflow.dflow INFO: Task 838 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6951065dc87249f7f05e141d1eabd822 with result from task 838 parsl.dataflow.dflow DEBUG: Task 838 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 839 has memoization hash bfaedf7286382bffb3269ea398538b4a parsl.dataflow.memoization INFO: Task 839 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45665080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 839 try 0 launched on executor multi with executor id 880 parsl.dataflow.dflow INFO: Standard output for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/d68242b7-d4a7-4b4a-9e67-2347927c5344_calibrate_855136_9.stdout parsl.dataflow.dflow INFO: Standard error for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/d68242b7-d4a7-4b4a-9e67-2347927c5344_calibrate_855136_9.stderr parsl.dataflow.dflow INFO: Standard output for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/46441930-63f7-430b-86f1-53c061a311e1_characterizeImage_855136_9.stdout parsl.dataflow.dflow INFO: Standard error for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/46441930-63f7-430b-86f1-53c061a311e1_characterizeImage_855136_9.stderr parsl.dataflow.dflow INFO: Task 916 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 833987b0c2882c73644cf35d553fb6cd with result from task 916 parsl.dataflow.dflow DEBUG: Task 916 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 917 has memoization hash 551beeb44f7de16d6a90a6a28f364758 parsl.dataflow.memoization INFO: Task 917 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4549a3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 917 try 0 launched on executor multi with executor id 881 parsl.dataflow.dflow INFO: Standard output for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/d8ca358d-c1dc-46c6-8765-28e394f0a26a_calibrate_855136_57.stdout parsl.dataflow.dflow INFO: Standard error for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/d8ca358d-c1dc-46c6-8765-28e394f0a26a_calibrate_855136_57.stderr parsl.dataflow.dflow INFO: Standard output for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/1ca73511-5813-4310-a233-081941d8b622_characterizeImage_855136_57.stdout parsl.dataflow.dflow INFO: Standard error for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/1ca73511-5813-4310-a233-081941d8b622_characterizeImage_855136_57.stderr parsl.dataflow.dflow INFO: Task 691 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03e1b7251e901b000d4e7bcd76c081f1 with result from task 691 parsl.dataflow.dflow DEBUG: Task 691 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 692 has memoization hash 8deb930f1b053404d421a7762ae554b4 parsl.dataflow.memoization INFO: Task 692 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 692 try 0 launched on executor multi with executor id 882 parsl.dataflow.dflow INFO: Standard output for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/173f3aae-4d0e-482d-ab51-4f779788cf61_calibrate_855136_20.stdout parsl.dataflow.dflow INFO: Standard error for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/173f3aae-4d0e-482d-ab51-4f779788cf61_calibrate_855136_20.stderr parsl.dataflow.dflow INFO: Standard output for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/fc0420c1-dc84-47f7-bc99-cb2dca208ea2_characterizeImage_855136_20.stdout parsl.dataflow.dflow INFO: Standard error for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/fc0420c1-dc84-47f7-bc99-cb2dca208ea2_characterizeImage_855136_20.stderr parsl.dataflow.dflow INFO: Task 749 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f34e23ac2c863527a0752d4dffde218 with result from task 749 parsl.dataflow.dflow DEBUG: Task 749 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/39c1c8c4-99e1-42f7-8bf7-c335fae945a1_calibrate_855136_31.stdout parsl.dataflow.dflow INFO: Standard error for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/39c1c8c4-99e1-42f7-8bf7-c335fae945a1_calibrate_855136_31.stderr parsl.dataflow.dflow INFO: Task 34 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2fa62125b2b362f1dbe20237dc66554e with result from task 34 parsl.dataflow.dflow DEBUG: Task 34 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 35 has memoization hash 1605addffc9d50970752dbdf7f6a3cb6 parsl.dataflow.memoization INFO: Task 35 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d04a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 35 try 0 launched on executor multi with executor id 883 parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/1028dae4-a45e-447e-800b-e0a7f158a0e1_calibrate_855137_7.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/1028dae4-a45e-447e-800b-e0a7f158a0e1_calibrate_855137_7.stderr parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/78263280-9d23-4733-aefc-ebeb93bdc068_characterizeImage_855137_7.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/78263280-9d23-4733-aefc-ebeb93bdc068_characterizeImage_855137_7.stderr parsl.dataflow.dflow INFO: Task 211 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 93fa0248c7c7cdcb8790de11515c59a3 with result from task 211 parsl.dataflow.dflow DEBUG: Task 211 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 212 has memoization hash 72c890fdf395158042e984d6c65eae5c parsl.dataflow.memoization INFO: Task 212 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46834040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 212 try 0 launched on executor multi with executor id 884 parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/f6a639d1-d57c-4194-af04-1c5749b8dafa_calibrate_855137_33.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/f6a639d1-d57c-4194-af04-1c5749b8dafa_calibrate_855137_33.stderr parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/6cc4cb45-3d86-4472-b71a-85186ee9aedb_characterizeImage_855137_33.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/6cc4cb45-3d86-4472-b71a-85186ee9aedb_characterizeImage_855137_33.stderr parsl.channels.local.local WARNING: Execution of command 'squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312'' failed due to Command 'squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312'' timed out after 10 seconds parsl.utils ERROR: Callback threw an exception - logging and proceeding anyway Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/utils.py", line 341, in make_callback self.callback(*self.cb_args) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/job_status_poller.py", line 114, in poll self._update_state() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/job_status_poller.py", line 121, in _update_state item.poll(now) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/job_status_poller.py", line 47, in poll self._status = self._executor.status() ^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/executors/status_handling.py", line 109, in status status = self._make_status_dict(block_ids, self._provider.status(job_ids)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/gscratch/dirac/shared/opt/proc_lsst/python/proc_lsst/multi.py", line 190, in status statuses += self.providers[provider].status([job_id]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/providers/cluster_provider.py", line 142, in status self._status() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/providers/slurm/slurm.py", line 157, in _status retcode, stdout, stderr = self.execute_wait(cmd) ^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/providers/cluster_provider.py", line 83, in execute_wait return self.channel.execute_wait(cmd, t) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/channels/local/local.py", line 71, in execute_wait (stdout, stderr) = proc.communicate(timeout=walltime) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/subprocess.py", line 1209, in communicate stdout, stderr = self._communicate(input, endtime, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/subprocess.py", line 2114, in _communicate self._check_timeout(endtime, orig_timeout, stdout, stderr) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/subprocess.py", line 1253, in _check_timeout raise TimeoutExpired( subprocess.TimeoutExpired: Command 'squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312'' timed out after 10 seconds 2024-03-08 07:18:32 proc_lsst.multi:146 [INFO] found job 75775 in provider local proc_lsst.multi INFO: found job 75775 in provider local parsl.dataflow.dflow INFO: Task 436 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d655dc835f9a48e141c3d929ad27c0c1 with result from task 436 parsl.dataflow.dflow DEBUG: Task 436 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 437 has memoization hash f97fc699ebf3eba0a8bb7712607f6f9f parsl.dataflow.memoization INFO: Task 437 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a4c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 437 try 0 launched on executor multi with executor id 885 parsl.dataflow.dflow INFO: Standard output for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/491b4b4d-32e3-4aa1-baa3-e1ac3d73b02f_calibrate_855137_14.stdout parsl.dataflow.dflow INFO: Standard error for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/491b4b4d-32e3-4aa1-baa3-e1ac3d73b02f_calibrate_855137_14.stderr parsl.dataflow.dflow INFO: Standard output for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/639f6a4e-87dc-4d83-93b8-dea0bafbadc6_characterizeImage_855137_14.stdout parsl.dataflow.dflow INFO: Standard error for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/639f6a4e-87dc-4d83-93b8-dea0bafbadc6_characterizeImage_855137_14.stderr 2024-03-08 07:18:32 proc_lsst.multi:146 [INFO] found job 16912283 in provider astro proc_lsst.multi INFO: found job 16912283 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 196 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8ed462420a3ef1096e500c842de4605 with result from task 196 parsl.dataflow.dflow DEBUG: Task 196 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 197 has memoization hash 583179ca727b52a599aa4cb6bb4e4ec4 parsl.dataflow.memoization INFO: Task 197 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46816f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 197 try 0 launched on executor multi with executor id 886 parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/2c28dbba-988a-44ca-8394-f21d0bf20496_calibrate_855136_45.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/2c28dbba-988a-44ca-8394-f21d0bf20496_calibrate_855136_45.stderr parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/69a4ba7d-f02c-4651-b1fc-99939d0131be_characterizeImage_855136_45.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/69a4ba7d-f02c-4651-b1fc-99939d0131be_characterizeImage_855136_45.stderr parsl.dataflow.dflow INFO: Task 403 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8a8b77e4699db74c29fbf3108fd5c80 with result from task 403 parsl.dataflow.dflow DEBUG: Task 403 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 404 has memoization hash b1921eb8435a473fc09f5b2893d95fa3 parsl.dataflow.memoization INFO: Task 404 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a79c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 404 try 0 launched on executor multi with executor id 887 parsl.dataflow.dflow INFO: Standard output for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/53d967a1-fa3b-4b0e-b685-8825365da314_calibrate_855136_43.stdout parsl.dataflow.dflow INFO: Standard error for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/53d967a1-fa3b-4b0e-b685-8825365da314_calibrate_855136_43.stderr parsl.dataflow.dflow INFO: Standard output for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/e91e187b-a720-4615-863e-9439f8e3cba9_characterizeImage_855136_43.stdout parsl.dataflow.dflow INFO: Standard error for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/e91e187b-a720-4615-863e-9439f8e3cba9_characterizeImage_855136_43.stderr parsl.dataflow.dflow INFO: Task 610 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91fadd832349dff6f5843dde51dc4995 with result from task 610 parsl.dataflow.dflow DEBUG: Task 610 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 611 has memoization hash e09da0234e4e4e06b55a92af458fc255 parsl.dataflow.memoization INFO: Task 611 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45741580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 611 try 0 launched on executor multi with executor id 888 parsl.dataflow.dflow INFO: Standard output for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/1203235c-f5ae-4a01-8b18-76caa38678c1_calibrate_855044_46.stdout parsl.dataflow.dflow INFO: Standard error for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/1203235c-f5ae-4a01-8b18-76caa38678c1_calibrate_855044_46.stderr parsl.dataflow.dflow INFO: Standard output for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/eecd9692-8df6-4407-a0ae-d5cd324a5cff_characterizeImage_855044_46.stdout parsl.dataflow.dflow INFO: Standard error for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/eecd9692-8df6-4407-a0ae-d5cd324a5cff_characterizeImage_855044_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:18:33 proc_lsst.multi:146 [INFO] found job 16912284 in provider astro proc_lsst.multi INFO: found job 16912284 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 85 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c118e3119241cb3db0960ff8e09fe49f with result from task 85 parsl.dataflow.dflow DEBUG: Task 85 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 86 has memoization hash f321e011ef317e49bca83236e5d4a1b7 parsl.dataflow.memoization INFO: Task 86 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d3b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 86 try 0 launched on executor multi with executor id 889 parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/4a67262a-7c79-4836-8acf-d390648b703a_calibrate_855137_21.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/4a67262a-7c79-4836-8acf-d390648b703a_calibrate_855137_21.stderr parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/54dc39bb-bdc0-4e4b-993e-2f1e038550f6_characterizeImage_855137_21.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/54dc39bb-bdc0-4e4b-993e-2f1e038550f6_characterizeImage_855137_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:18:34 proc_lsst.multi:146 [INFO] found job 16912285 in provider astro proc_lsst.multi INFO: found job 16912285 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 220 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b94649128ddc57d620c00367ceebbb9 with result from task 220 parsl.dataflow.dflow DEBUG: Task 220 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 221 has memoization hash ad8d5c449595bc19262aa3a3de050a1a parsl.dataflow.memoization INFO: Task 221 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46834860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 221 try 0 launched on executor multi with executor id 890 parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/911d9d41-4815-4487-afa8-d011c9437ab0_calibrate_855043_32.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/911d9d41-4815-4487-afa8-d011c9437ab0_calibrate_855043_32.stderr parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/10aa27db-ff53-4f11-9f19-9709837bafe7_characterizeImage_855043_32.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/10aa27db-ff53-4f11-9f19-9709837bafe7_characterizeImage_855043_32.stderr parsl.dataflow.dflow INFO: Task 592 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9cb5010a0bec05b8824b186945ab83b7 with result from task 592 parsl.dataflow.dflow DEBUG: Task 592 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 593 has memoization hash 357f490e236c70c55819cf542b319e22 parsl.dataflow.memoization INFO: Task 593 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45742a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 593 try 0 launched on executor multi with executor id 891 parsl.dataflow.dflow INFO: Standard output for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/3ca973ff-378e-44bd-bc49-f3798086fb4e_calibrate_855138_43.stdout parsl.dataflow.dflow INFO: Standard error for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/3ca973ff-378e-44bd-bc49-f3798086fb4e_calibrate_855138_43.stderr parsl.dataflow.dflow INFO: Standard output for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/6dc1d204-da4f-4d16-9ae3-f8c7f465049a_characterizeImage_855138_43.stdout parsl.dataflow.dflow INFO: Standard error for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/6dc1d204-da4f-4d16-9ae3-f8c7f465049a_characterizeImage_855138_43.stderr parsl.dataflow.dflow INFO: Task 703 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be531a8df134e094a32859e6f261d73c with result from task 703 parsl.dataflow.dflow DEBUG: Task 703 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 704 has memoization hash 81842c535ff8a638920773e56454da36 parsl.dataflow.memoization INFO: Task 704 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d0540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 704 try 0 launched on executor multi with executor id 892 parsl.dataflow.dflow INFO: Standard output for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/1a168c20-3169-498b-bda7-1ac0f54b172c_calibrate_855137_46.stdout parsl.dataflow.dflow INFO: Standard error for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/1a168c20-3169-498b-bda7-1ac0f54b172c_calibrate_855137_46.stderr parsl.dataflow.dflow INFO: Standard output for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/f22f06d8-3813-497a-b4c5-cd72dfa33f17_characterizeImage_855137_46.stdout parsl.dataflow.dflow INFO: Standard error for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/f22f06d8-3813-497a-b4c5-cd72dfa33f17_characterizeImage_855137_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:18:34 proc_lsst.multi:146 [INFO] found job 16912286 in provider astro proc_lsst.multi INFO: found job 16912286 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:18:35 proc_lsst.multi:146 [INFO] found job 16912287 in provider astro proc_lsst.multi INFO: found job 16912287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 751 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3d73041bd71b12124a9e9355188758c with result from task 751 parsl.dataflow.dflow DEBUG: Task 751 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 752 has memoization hash adcf5eb84dea58b461e4f535eb28a1d1 parsl.dataflow.memoization INFO: Task 752 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d2840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 752 try 0 launched on executor multi with executor id 893 parsl.dataflow.dflow INFO: Standard output for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/971aff00-e271-4961-8bdd-2057d995c57b_calibrate_855138_44.stdout parsl.dataflow.dflow INFO: Standard error for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/971aff00-e271-4961-8bdd-2057d995c57b_calibrate_855138_44.stderr parsl.dataflow.dflow INFO: Standard output for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/9702e711-d34d-4709-ad99-42b9d440be0b_characterizeImage_855138_44.stdout parsl.dataflow.dflow INFO: Standard error for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/9702e711-d34d-4709-ad99-42b9d440be0b_characterizeImage_855138_44.stderr parsl.dataflow.dflow INFO: Task 463 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c37137196978f3565a92737c00d00371 with result from task 463 parsl.dataflow.dflow DEBUG: Task 463 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 464 has memoization hash 3bdfa8f1440ff8d494875bf4b453c7d8 parsl.dataflow.memoization INFO: Task 464 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456cf9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 464 try 0 launched on executor multi with executor id 894 parsl.dataflow.dflow INFO: Standard output for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/015bb548-2c65-4c6a-b486-99522e1e4747_calibrate_855136_40.stdout parsl.dataflow.dflow INFO: Standard error for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/015bb548-2c65-4c6a-b486-99522e1e4747_calibrate_855136_40.stderr parsl.dataflow.dflow INFO: Standard output for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/217395bd-5bc4-4461-9c6e-b3b302002d85_characterizeImage_855136_40.stdout parsl.dataflow.dflow INFO: Standard error for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/217395bd-5bc4-4461-9c6e-b3b302002d85_characterizeImage_855136_40.stderr parsl.dataflow.dflow INFO: Task 244 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e376ac8f10530129fd53efd2a6401d55 with result from task 244 parsl.dataflow.dflow DEBUG: Task 244 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 245 has memoization hash 2121a71fb26ddaa807e4da7f4a088a52 parsl.dataflow.memoization INFO: Task 245 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46836160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 245 try 0 launched on executor multi with executor id 895 parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/5d6661e9-5552-4b7b-9830-ce2b5bcdda3f_calibrate_855043_13.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/5d6661e9-5552-4b7b-9830-ce2b5bcdda3f_calibrate_855043_13.stderr parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/cecac4d3-2ee8-4834-b131-4e489d30b935_characterizeImage_855043_13.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/cecac4d3-2ee8-4834-b131-4e489d30b935_characterizeImage_855043_13.stderr parsl.dataflow.dflow INFO: Task 259 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7910c794b8413b9641611af7e0db07b with result from task 259 parsl.dataflow.dflow DEBUG: Task 259 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 260 has memoization hash 6b53bb93043ee931943088df609d31bc parsl.dataflow.memoization INFO: Task 260 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46837100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 260 try 0 launched on executor multi with executor id 896 parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/6d64b5ce-7116-4855-881c-c7588f582d9b_calibrate_855137_5.stdout parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/6d64b5ce-7116-4855-881c-c7588f582d9b_calibrate_855137_5.stderr parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/27b7841f-221f-471b-8900-4c0054eb0c96_characterizeImage_855137_5.stdout parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/27b7841f-221f-471b-8900-4c0054eb0c96_characterizeImage_855137_5.stderr parsl.dataflow.dflow INFO: Task 451 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4feacb6f2a4122e107874f4dc8969f9 with result from task 451 parsl.dataflow.dflow DEBUG: Task 451 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 452 has memoization hash d250e93a816b13c6939d35ffb52c515d parsl.dataflow.memoization INFO: Task 452 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a4220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 452 try 0 launched on executor multi with executor id 897 parsl.dataflow.dflow INFO: Standard output for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/c1d3eb70-34ea-463d-8305-b54f531508ad_calibrate_855138_9.stdout parsl.dataflow.dflow INFO: Standard error for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/c1d3eb70-34ea-463d-8305-b54f531508ad_calibrate_855138_9.stderr parsl.dataflow.dflow INFO: Standard output for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/c18d4c69-f7da-4493-ba19-0a9ba05a7692_characterizeImage_855138_9.stdout parsl.dataflow.dflow INFO: Standard error for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/c18d4c69-f7da-4493-ba19-0a9ba05a7692_characterizeImage_855138_9.stderr parsl.dataflow.dflow INFO: Task 1051 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5faaae8cd178d7e08533d52c49334cb with result from task 1051 parsl.dataflow.dflow DEBUG: Task 1051 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1052 has memoization hash 3b57fdee9c16ad7465a730e15a5dd77d parsl.dataflow.memoization INFO: Task 1052 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45543100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1052 try 0 launched on executor multi with executor id 898 parsl.dataflow.dflow INFO: Standard output for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/a6ba4420-61de-4182-b505-a8a2996a099b_calibrate_855043_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/a6ba4420-61de-4182-b505-a8a2996a099b_calibrate_855043_51.stderr parsl.dataflow.dflow INFO: Standard output for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/884c8170-c1f3-4702-9490-fc40690cd237_characterizeImage_855043_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/884c8170-c1f3-4702-9490-fc40690cd237_characterizeImage_855043_51.stderr parsl.dataflow.dflow INFO: Task 241 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97b7a7c494fc5dcab8a98ec6fa8cd47e with result from task 241 parsl.dataflow.dflow DEBUG: Task 241 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 242 has memoization hash e06b8afde1b1d431728f48599c937304 parsl.dataflow.memoization INFO: Task 242 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46835e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 242 try 0 launched on executor multi with executor id 899 parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/a13abcfe-b214-4dbb-94bc-a5aaad76570b_calibrate_855136_8.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/a13abcfe-b214-4dbb-94bc-a5aaad76570b_calibrate_855136_8.stderr parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/a461e955-6651-4a33-9764-d79444da84f4_characterizeImage_855136_8.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/a461e955-6651-4a33-9764-d79444da84f4_characterizeImage_855136_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:18:39 proc_lsst.multi:146 [INFO] found job 16912288 in provider astro proc_lsst.multi INFO: found job 16912288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 395 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68a3fe6c87fcfe259c899f4859fc5de4 with result from task 395 parsl.dataflow.dflow DEBUG: Task 395 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/9bd703e0-ba48-46ba-98db-3b9193da9953_calibrate_855044_5.stdout parsl.dataflow.dflow INFO: Standard error for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/9bd703e0-ba48-46ba-98db-3b9193da9953_calibrate_855044_5.stderr parsl.dataflow.dflow INFO: Task 43 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry efd662c842b6ea42999b0e0bee126020 with result from task 43 parsl.dataflow.dflow DEBUG: Task 43 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 44 has memoization hash 7019fbbb186cba8d2fb243d4c90cf829 parsl.dataflow.memoization INFO: Task 44 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d0b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 44 try 0 launched on executor multi with executor id 900 parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/de8e9723-7dcd-49e4-861d-754e99cd4eaf_calibrate_855136_35.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/de8e9723-7dcd-49e4-861d-754e99cd4eaf_calibrate_855136_35.stderr parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/57ff7d87-8c9c-47a7-b148-25d7985035a6_characterizeImage_855136_35.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/57ff7d87-8c9c-47a7-b148-25d7985035a6_characterizeImage_855136_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:18:40 proc_lsst.multi:146 [INFO] found job 16912289 in provider astro proc_lsst.multi INFO: found job 16912289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 22 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96f2f2e1e0d4b57f9ebcaed8a43faaa5 with result from task 22 parsl.dataflow.dflow DEBUG: Task 22 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 23 has memoization hash 9b5ab7932944601e606ab36feca69e32 parsl.dataflow.memoization INFO: Task 23 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467b37e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 23 try 0 launched on executor multi with executor id 901 parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/2f40fd4f-1aaa-48a8-adf4-6c1c48f251b0_calibrate_855137_52.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/2f40fd4f-1aaa-48a8-adf4-6c1c48f251b0_calibrate_855137_52.stderr parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/6915b873-8256-46f0-8d36-a7f567d3536d_characterizeImage_855137_52.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/6915b873-8256-46f0-8d36-a7f567d3536d_characterizeImage_855137_52.stderr parsl.dataflow.dflow INFO: Task 587 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 463bee33dd3ba10d0e11bbb399065ca4 with result from task 587 parsl.dataflow.dflow DEBUG: Task 587 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/46bc6e0f-d910-49f4-9b22-1432a12bac1e_calibrate_855042_21.stdout parsl.dataflow.dflow INFO: Standard error for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/46bc6e0f-d910-49f4-9b22-1432a12bac1e_calibrate_855042_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:18:40 proc_lsst.multi:146 [INFO] found job 16912290 in provider astro proc_lsst.multi INFO: found job 16912290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 889 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b3a1ead4a703b60c36d862e21c472d90 with result from task 889 parsl.dataflow.dflow DEBUG: Task 889 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 890 has memoization hash ed35f1563f192bad6e19c9cea28fa6d8 parsl.dataflow.memoization INFO: Task 890 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45667ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 890 try 0 launched on executor multi with executor id 902 parsl.dataflow.dflow INFO: Standard output for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/2473ae04-b3fe-4f3d-9f3d-5628af1c1084_calibrate_855043_52.stdout parsl.dataflow.dflow INFO: Standard error for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/2473ae04-b3fe-4f3d-9f3d-5628af1c1084_calibrate_855043_52.stderr parsl.dataflow.dflow INFO: Standard output for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/3158853d-2c8e-4d8e-860e-b18c3b2baf82_characterizeImage_855043_52.stdout parsl.dataflow.dflow INFO: Standard error for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/3158853d-2c8e-4d8e-860e-b18c3b2baf82_characterizeImage_855043_52.stderr parsl.dataflow.dflow INFO: Task 46 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f0c6239bea39b99ae448343c1e760c0 with result from task 46 parsl.dataflow.dflow DEBUG: Task 46 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 47 has memoization hash 118a473971fe6ca82da4bdbf54320dc7 parsl.dataflow.memoization INFO: Task 47 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d1260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 47 try 0 launched on executor multi with executor id 903 parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/3cc55836-d648-4972-b6ac-b9e6f647900d_calibrate_855138_32.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/3cc55836-d648-4972-b6ac-b9e6f647900d_calibrate_855138_32.stderr parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/200eeb73-9f08-4d6a-a7d6-ab66fb193ebd_characterizeImage_855138_32.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/200eeb73-9f08-4d6a-a7d6-ab66fb193ebd_characterizeImage_855138_32.stderr parsl.dataflow.dflow INFO: Task 904 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c40e3de881b69a96ee6846378b43b902 with result from task 904 parsl.dataflow.dflow DEBUG: Task 904 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 905 has memoization hash d9ac90ff4cade1779b5c802c71192dc0 parsl.dataflow.memoization INFO: Task 905 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454982c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 905 try 0 launched on executor multi with executor id 904 parsl.dataflow.dflow INFO: Standard output for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/db5a35f7-47d6-404e-9c5a-39f321faf180_calibrate_855043_56.stdout parsl.dataflow.dflow INFO: Standard error for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/db5a35f7-47d6-404e-9c5a-39f321faf180_calibrate_855043_56.stderr parsl.dataflow.dflow INFO: Standard output for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/7fd5cf92-1dbe-4bbc-b1b7-770ace5ff059_characterizeImage_855043_56.stdout parsl.dataflow.dflow INFO: Standard error for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/7fd5cf92-1dbe-4bbc-b1b7-770ace5ff059_characterizeImage_855043_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:18:41 proc_lsst.multi:146 [INFO] found job 16912291 in provider astro proc_lsst.multi INFO: found job 16912291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 922 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ab9dc8f962eafb6e49e897bb9a8d50d with result from task 922 parsl.dataflow.dflow DEBUG: Task 922 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 923 has memoization hash 2d083ab974a92d68576ff12c39e1e523 parsl.dataflow.memoization INFO: Task 923 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4549aa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 923 try 0 launched on executor multi with executor id 905 parsl.dataflow.dflow INFO: Standard output for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/915e5395-6835-4105-899f-fd2cba696be1_calibrate_855044_55.stdout parsl.dataflow.dflow INFO: Standard error for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/915e5395-6835-4105-899f-fd2cba696be1_calibrate_855044_55.stderr parsl.dataflow.dflow INFO: Standard output for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/344e994b-7936-421f-b2b3-7b242ac710ba_characterizeImage_855044_55.stdout parsl.dataflow.dflow INFO: Standard error for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/344e994b-7936-421f-b2b3-7b242ac710ba_characterizeImage_855044_55.stderr parsl.dataflow.dflow INFO: Task 265 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cafcf8df6ccc1b224928b89886eecd95 with result from task 265 parsl.dataflow.dflow DEBUG: Task 265 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 266 has memoization hash 24cc5243d6d479c062ffdfe65b249a9a parsl.dataflow.memoization INFO: Task 266 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46837740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 266 try 0 launched on executor multi with executor id 906 parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/b82ce910-d779-489e-be01-31f7f39d10b5_calibrate_855044_40.stdout parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/b82ce910-d779-489e-be01-31f7f39d10b5_calibrate_855044_40.stderr parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/d113dfd3-beb2-40bb-b845-771b99a21647_characterizeImage_855044_40.stdout parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/d113dfd3-beb2-40bb-b845-771b99a21647_characterizeImage_855044_40.stderr parsl.dataflow.dflow INFO: Task 574 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7b008856ddd413dbf03ad72b12fffe4 with result from task 574 parsl.dataflow.dflow DEBUG: Task 574 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 575 has memoization hash f34e4f4b6db1ebf96801c1f1aba765e3 parsl.dataflow.memoization INFO: Task 575 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45700cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 575 try 0 launched on executor multi with executor id 907 parsl.dataflow.dflow INFO: Standard output for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/580c67e8-88f5-4073-a324-52ad94f043f5_calibrate_855138_6.stdout parsl.dataflow.dflow INFO: Standard error for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/580c67e8-88f5-4073-a324-52ad94f043f5_calibrate_855138_6.stderr parsl.dataflow.dflow INFO: Standard output for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/d7dac109-b10f-437b-a91d-3cc7ded677f2_characterizeImage_855138_6.stdout parsl.dataflow.dflow INFO: Standard error for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/d7dac109-b10f-437b-a91d-3cc7ded677f2_characterizeImage_855138_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:18:42 proc_lsst.multi:146 [INFO] found job 16912292 in provider astro proc_lsst.multi INFO: found job 16912292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 16 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a0d63fbd6f10d9cad2a14e70cbb4a37 with result from task 16 parsl.dataflow.dflow DEBUG: Task 16 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 17 has memoization hash 7dcd3cb6421e4a3506663f59b0ba5d1c parsl.dataflow.memoization INFO: Task 17 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467b1f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 17 try 0 launched on executor multi with executor id 908 parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/c9157d28-0381-4d0c-a20c-2d8b608945fb_calibrate_855044_53.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/c9157d28-0381-4d0c-a20c-2d8b608945fb_calibrate_855044_53.stderr parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/1d5302e0-40c6-4288-92ed-fb0d299d3f72_characterizeImage_855044_53.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/1d5302e0-40c6-4288-92ed-fb0d299d3f72_characterizeImage_855044_53.stderr parsl.dataflow.dflow INFO: Task 331 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fcaea86c80afbd0603f3ae2d16834512 with result from task 331 parsl.dataflow.dflow DEBUG: Task 331 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 332 has memoization hash 64bf483b735baf23a1456faa55428608 parsl.dataflow.memoization INFO: Task 332 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 332 try 0 launched on executor multi with executor id 909 parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/fb05422e-4e85-4dfc-8e66-3fe74529f27d_calibrate_855043_62.stdout parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/fb05422e-4e85-4dfc-8e66-3fe74529f27d_calibrate_855043_62.stderr parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/6bf80558-1bee-41cc-b8bf-bade6a0f2df4_characterizeImage_855043_62.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/6bf80558-1bee-41cc-b8bf-bade6a0f2df4_characterizeImage_855043_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:18:43 proc_lsst.multi:146 [INFO] found job 16912293 in provider astro proc_lsst.multi INFO: found job 16912293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 286 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2bc3e5100576f3a1f83bb4d2c5925ef6 with result from task 286 parsl.dataflow.dflow DEBUG: Task 286 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 287 has memoization hash 66df182438cc20212da5d4b36952c838 parsl.dataflow.memoization INFO: Task 287 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 287 try 0 launched on executor multi with executor id 910 parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/698e6f7d-c2e9-49f5-b39a-8bdef912dd9d_calibrate_855137_29.stdout parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/698e6f7d-c2e9-49f5-b39a-8bdef912dd9d_calibrate_855137_29.stderr parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/b8464340-e866-40dc-b19c-af5cfecf2950_characterizeImage_855137_29.stdout parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/b8464340-e866-40dc-b19c-af5cfecf2950_characterizeImage_855137_29.stderr parsl.dataflow.dflow INFO: Task 127 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce5ca66c91dc6fc1decfb230c443e68a with result from task 127 parsl.dataflow.dflow DEBUG: Task 127 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 128 has memoization hash fba1ce37e6808ada60f92ad403a2ee46 parsl.dataflow.memoization INFO: Task 128 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467f6700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 128 try 0 launched on executor multi with executor id 911 parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/fc13d0dc-b5b1-4fde-a3f6-684626c2e7aa_calibrate_855136_42.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/fc13d0dc-b5b1-4fde-a3f6-684626c2e7aa_calibrate_855136_42.stderr parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/ff07f600-431e-42df-bc97-33ba2d86a5f1_characterizeImage_855136_42.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/ff07f600-431e-42df-bc97-33ba2d86a5f1_characterizeImage_855136_42.stderr parsl.dataflow.dflow INFO: Task 223 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10753c3c50ecaac721e4da8d5680e07a with result from task 223 parsl.dataflow.dflow DEBUG: Task 223 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 224 has memoization hash d1a200adad3a8a7e66fd101c3f4f5e5b parsl.dataflow.memoization INFO: Task 224 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46834b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 224 try 0 launched on executor multi with executor id 912 parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/f44d25b0-27ae-46eb-ad72-97d2639831ca_calibrate_855044_32.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/f44d25b0-27ae-46eb-ad72-97d2639831ca_calibrate_855044_32.stderr parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/63a5eb83-4b2f-4c56-9880-64ccddeef7a2_characterizeImage_855044_32.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/63a5eb83-4b2f-4c56-9880-64ccddeef7a2_characterizeImage_855044_32.stderr parsl.dataflow.dflow INFO: Task 271 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96b98422f8159adc65593fb1bb06a7d7 with result from task 271 parsl.dataflow.dflow DEBUG: Task 271 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 272 has memoization hash c7b12b0725cd0abe9a3f3515d1887e18 parsl.dataflow.memoization INFO: Task 272 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46837d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 272 try 0 launched on executor multi with executor id 913 parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/ab5b17b7-2f19-4c9f-9775-1f774a10b55c_calibrate_855042_16.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/ab5b17b7-2f19-4c9f-9775-1f774a10b55c_calibrate_855042_16.stderr parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/ef65dffc-6a37-4620-b178-4caa1c058ea3_characterizeImage_855042_16.stdout parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/ef65dffc-6a37-4620-b178-4caa1c058ea3_characterizeImage_855042_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:18:43 proc_lsst.multi:146 [INFO] found job 16912294 in provider astro proc_lsst.multi INFO: found job 16912294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 184 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 738c5b454ac0816ea11364c9ceeeeae2 with result from task 184 parsl.dataflow.dflow DEBUG: Task 184 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 185 has memoization hash 43e179cb8f657e8cee64d8ddf3202595 parsl.dataflow.memoization INFO: Task 185 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa468162a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 185 try 0 launched on executor multi with executor id 914 parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/b3dab249-30f0-472d-859b-9f30821f05db_calibrate_855137_6.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/b3dab249-30f0-472d-859b-9f30821f05db_calibrate_855137_6.stderr parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/edeadeec-41b9-44c8-9832-6740d2ce423e_characterizeImage_855137_6.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/edeadeec-41b9-44c8-9832-6740d2ce423e_characterizeImage_855137_6.stderr parsl.dataflow.dflow INFO: Task 226 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9551be45c8e136c9616da13a257d673 with result from task 226 parsl.dataflow.dflow DEBUG: Task 226 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 227 has memoization hash 4e2399e2b58072ee79dabbe1eee80fe1 parsl.dataflow.memoization INFO: Task 227 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46834ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 227 try 0 launched on executor multi with executor id 915 parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/f6ddf3a4-6fdc-49cf-b96c-e252f3e6faae_calibrate_855137_48.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/f6ddf3a4-6fdc-49cf-b96c-e252f3e6faae_calibrate_855137_48.stderr parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/de125ba2-dcc8-47be-bc99-d9e833a1da6d_characterizeImage_855137_48.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/de125ba2-dcc8-47be-bc99-d9e833a1da6d_characterizeImage_855137_48.stderr parsl.dataflow.dflow INFO: Task 991 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78f4118382e357fe9ddb28bd3de3ec88 with result from task 991 parsl.dataflow.dflow DEBUG: Task 991 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 992 has memoization hash 9b20541a6347cdc28fb1e83c8106fa28 parsl.dataflow.memoization INFO: Task 992 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454fb240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 992 try 0 launched on executor multi with executor id 916 parsl.dataflow.dflow INFO: Standard output for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/9e9ec239-02ec-4d81-8a34-51e5eb27744e_calibrate_855044_11.stdout parsl.dataflow.dflow INFO: Standard error for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/9e9ec239-02ec-4d81-8a34-51e5eb27744e_calibrate_855044_11.stderr parsl.dataflow.dflow INFO: Standard output for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/a29d6546-3790-4c37-abb1-17e5e204d174_characterizeImage_855044_11.stdout parsl.dataflow.dflow INFO: Standard error for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/a29d6546-3790-4c37-abb1-17e5e204d174_characterizeImage_855044_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:18:44 proc_lsst.multi:146 [INFO] found job 16912295 in provider astro proc_lsst.multi INFO: found job 16912295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 758 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf03690391893b3ad16f8ccd3d6545b0 with result from task 758 parsl.dataflow.dflow DEBUG: Task 758 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/514a49f3-60ba-44ed-afa6-af0c5d5416bd_calibrate_855044_10.stdout parsl.dataflow.dflow INFO: Standard error for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/514a49f3-60ba-44ed-afa6-af0c5d5416bd_calibrate_855044_10.stderr parsl.dataflow.dflow INFO: Task 571 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d51d33fab9285dd100640c93e94f2e0d with result from task 571 parsl.dataflow.dflow DEBUG: Task 571 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 572 has memoization hash 7c747fec1799467ebe099e7b944183ff parsl.dataflow.memoization INFO: Task 572 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa457031a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 572 try 0 launched on executor multi with executor id 917 parsl.dataflow.dflow INFO: Standard output for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/9361bbd5-c16c-42f8-bd22-98c4bfce5474_calibrate_855043_53.stdout parsl.dataflow.dflow INFO: Standard error for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/9361bbd5-c16c-42f8-bd22-98c4bfce5474_calibrate_855043_53.stderr parsl.dataflow.dflow INFO: Standard output for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/6662812e-ff40-4ff7-9759-8f7e94d0c502_characterizeImage_855043_53.stdout parsl.dataflow.dflow INFO: Standard error for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/6662812e-ff40-4ff7-9759-8f7e94d0c502_characterizeImage_855043_53.stderr parsl.dataflow.dflow INFO: Task 845 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42584de13e0ba9c74cfadc2648ca97d7 with result from task 845 parsl.dataflow.dflow DEBUG: Task 845 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/09a864ff-b53b-45e8-9593-f1c4fadad042_calibrate_855043_40.stdout parsl.dataflow.dflow INFO: Standard error for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/09a864ff-b53b-45e8-9593-f1c4fadad042_calibrate_855043_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:18:45 proc_lsst.multi:146 [INFO] found job 16912296 in provider astro proc_lsst.multi INFO: found job 16912296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 799 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2cc0e6a66d9d6e52577586f3255f7d0a with result from task 799 parsl.dataflow.dflow DEBUG: Task 799 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 800 has memoization hash 83265f3d5bba8b78c2a470334aa9cba0 parsl.dataflow.memoization INFO: Task 800 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 800 try 0 launched on executor multi with executor id 918 parsl.dataflow.dflow INFO: Standard output for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/c35c0c7b-9cd0-4e6e-aca7-4951b1282f2f_calibrate_855138_35.stdout parsl.dataflow.dflow INFO: Standard error for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/c35c0c7b-9cd0-4e6e-aca7-4951b1282f2f_calibrate_855138_35.stderr parsl.dataflow.dflow INFO: Standard output for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/75ba3708-493d-4a90-b006-a63d1c11ceb9_characterizeImage_855138_35.stdout parsl.dataflow.dflow INFO: Standard error for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/75ba3708-493d-4a90-b006-a63d1c11ceb9_characterizeImage_855138_35.stderr parsl.dataflow.dflow INFO: Task 584 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 424e677337099e6b9560db704a53e67c with result from task 584 parsl.dataflow.dflow DEBUG: Task 584 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/b1d9c744-6fe0-4880-a5b5-e4e5ebb1e041_calibrate_855042_57.stdout parsl.dataflow.dflow INFO: Standard error for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/b1d9c744-6fe0-4880-a5b5-e4e5ebb1e041_calibrate_855042_57.stderr parsl.dataflow.dflow INFO: Task 373 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ef98ff2eb21a4736bf70f10ee3245e4 with result from task 373 parsl.dataflow.dflow DEBUG: Task 373 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 374 has memoization hash e25366e944f6df83ddcc3f03fb3e4056 parsl.dataflow.memoization INFO: Task 374 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4687a840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 374 try 0 launched on executor multi with executor id 919 parsl.dataflow.dflow INFO: Standard output for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/094bc063-3328-43d0-afca-b0cc248491b1_calibrate_855043_47.stdout parsl.dataflow.dflow INFO: Standard error for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/094bc063-3328-43d0-afca-b0cc248491b1_calibrate_855043_47.stderr parsl.dataflow.dflow INFO: Standard output for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/6023be9b-6ea4-43d6-8d6e-6f0b1f7ed3f5_characterizeImage_855043_47.stdout parsl.dataflow.dflow INFO: Standard error for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/6023be9b-6ea4-43d6-8d6e-6f0b1f7ed3f5_characterizeImage_855043_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:18:47 proc_lsst.multi:146 [INFO] found job 16912297 in provider astro proc_lsst.multi INFO: found job 16912297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 475 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1891a1cb03115b8a28c6b6a16f2efdd2 with result from task 475 parsl.dataflow.dflow DEBUG: Task 475 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 476 has memoization hash 466a67c30f95792bbaed5aaa4a5b8475 parsl.dataflow.memoization INFO: Task 476 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456ced40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 476 try 0 launched on executor multi with executor id 920 parsl.dataflow.dflow INFO: Standard output for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/d7b8bd24-d2bc-41f1-9836-09c59aa44fb1_calibrate_855043_14.stdout parsl.dataflow.dflow INFO: Standard error for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/d7b8bd24-d2bc-41f1-9836-09c59aa44fb1_calibrate_855043_14.stderr parsl.dataflow.dflow INFO: Standard output for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/9a381784-5938-4b22-a11c-85d1a7f61e3d_characterizeImage_855043_14.stdout parsl.dataflow.dflow INFO: Standard error for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/9a381784-5938-4b22-a11c-85d1a7f61e3d_characterizeImage_855043_14.stderr parsl.dataflow.dflow INFO: Task 446 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8b7c8accc7eff676b01b3ea68406bf6 with result from task 446 parsl.dataflow.dflow DEBUG: Task 446 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/5f50344f-c795-4405-a281-40c8d875ad80_calibrate_855043_1.stdout parsl.dataflow.dflow INFO: Standard error for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/5f50344f-c795-4405-a281-40c8d875ad80_calibrate_855043_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:18:48 proc_lsst.multi:146 [INFO] found job 16912298 in provider astro proc_lsst.multi INFO: found job 16912298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 983 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 40b70bcec417858a96b30b3573ea8347 with result from task 983 parsl.dataflow.dflow DEBUG: Task 983 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/93b0e79b-3cad-4152-856e-1ac830cc0731_calibrate_855138_31.stdout parsl.dataflow.dflow INFO: Standard error for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/93b0e79b-3cad-4152-856e-1ac830cc0731_calibrate_855138_31.stderr parsl.dataflow.dflow INFO: Task 454 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cfa527e6a461d6dd7b848cb12fe702fa with result from task 454 parsl.dataflow.dflow DEBUG: Task 454 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 455 has memoization hash 3b99bde04c8ed542b8727fee68b6c5e6 parsl.dataflow.memoization INFO: Task 455 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a7d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 455 try 0 launched on executor multi with executor id 921 parsl.dataflow.dflow INFO: Standard output for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/ea03c0c0-da17-4c8c-a881-2187f74efe12_calibrate_855137_20.stdout parsl.dataflow.dflow INFO: Standard error for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/ea03c0c0-da17-4c8c-a881-2187f74efe12_calibrate_855137_20.stderr parsl.dataflow.dflow INFO: Standard output for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/bb1224a5-aff1-40fd-9d5d-ac6d850b924f_characterizeImage_855137_20.stdout parsl.dataflow.dflow INFO: Standard error for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/bb1224a5-aff1-40fd-9d5d-ac6d850b924f_characterizeImage_855137_20.stderr parsl.dataflow.dflow INFO: Task 577 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f91764a5e09d442e62b2f2f9cdd420b1 with result from task 577 parsl.dataflow.dflow DEBUG: Task 577 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 578 has memoization hash e66533046e073da9bf547d29af23d9d3 parsl.dataflow.memoization INFO: Task 578 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45740220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 578 try 0 launched on executor multi with executor id 922 parsl.dataflow.dflow INFO: Standard output for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/57a4c4c2-98fc-4c5f-b84f-9023d1a3730a_calibrate_855044_18.stdout parsl.dataflow.dflow INFO: Standard error for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/57a4c4c2-98fc-4c5f-b84f-9023d1a3730a_calibrate_855044_18.stderr parsl.dataflow.dflow INFO: Standard output for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/e8e15fd7-29b9-4789-ae3d-21b93a64ec0c_characterizeImage_855044_18.stdout parsl.dataflow.dflow INFO: Standard error for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/e8e15fd7-29b9-4789-ae3d-21b93a64ec0c_characterizeImage_855044_18.stderr parsl.dataflow.dflow INFO: Task 652 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7763ec8ffa4d72f3f38b17b5a07a4c0b with result from task 652 parsl.dataflow.dflow DEBUG: Task 652 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 653 has memoization hash 4e981ec66bd824c44c6eb6bb559d79f0 parsl.dataflow.memoization INFO: Task 653 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 653 try 0 launched on executor multi with executor id 923 parsl.dataflow.dflow INFO: Standard output for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/e244a8b4-d1e9-4c2a-8a4f-376fd4861b95_calibrate_855042_9.stdout parsl.dataflow.dflow INFO: Standard error for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/e244a8b4-d1e9-4c2a-8a4f-376fd4861b95_calibrate_855042_9.stderr parsl.dataflow.dflow INFO: Standard output for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/89538e24-b316-4485-94df-162c768c72f1_characterizeImage_855042_9.stdout parsl.dataflow.dflow INFO: Standard error for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/89538e24-b316-4485-94df-162c768c72f1_characterizeImage_855042_9.stderr parsl.dataflow.dflow INFO: Task 193 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 26ac35739269d2ec230b77897efc0845 with result from task 193 parsl.dataflow.dflow DEBUG: Task 193 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 194 has memoization hash a2e63d903bde3d4d65f2f238d86db5f8 parsl.dataflow.memoization INFO: Task 194 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46816c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 194 try 0 launched on executor multi with executor id 924 parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/147501f0-f2cb-4d87-a1a5-5c0d45d65d97_calibrate_855136_33.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/147501f0-f2cb-4d87-a1a5-5c0d45d65d97_calibrate_855136_33.stderr parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/2b849466-3b1a-41a0-9353-96411c953838_characterizeImage_855136_33.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/2b849466-3b1a-41a0-9353-96411c953838_characterizeImage_855136_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:18:50 proc_lsst.multi:146 [INFO] found job 16912299 in provider astro proc_lsst.multi INFO: found job 16912299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 214 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 245fd318e300db532a878985314edde1 with result from task 214 parsl.dataflow.dflow DEBUG: Task 214 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 215 has memoization hash a5b5eba8c032672117f1d561c12c8583 parsl.dataflow.memoization INFO: Task 215 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46834400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 215 try 0 launched on executor multi with executor id 925 parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/fdbd57a6-7884-445c-bd5d-839b42feabba_calibrate_855043_12.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/fdbd57a6-7884-445c-bd5d-839b42feabba_calibrate_855043_12.stderr parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/e4892757-a874-46b3-ba71-56277a905abf_characterizeImage_855043_12.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/e4892757-a874-46b3-ba71-56277a905abf_characterizeImage_855043_12.stderr parsl.dataflow.dflow INFO: Task 151 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb67acbd45604b457b27824259626780 with result from task 151 parsl.dataflow.dflow DEBUG: Task 151 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 152 has memoization hash 1bbc24de45724b797f7e7fcb6c57a798 parsl.dataflow.memoization INFO: Task 152 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46814040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 152 try 0 launched on executor multi with executor id 926 parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/2c0a2bf8-00d4-4ac7-947b-1b17da00833b_calibrate_855043_29.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/2c0a2bf8-00d4-4ac7-947b-1b17da00833b_calibrate_855043_29.stderr parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/65fd74c5-4005-4161-9163-db77bd4a52f4_characterizeImage_855043_29.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/65fd74c5-4005-4161-9163-db77bd4a52f4_characterizeImage_855043_29.stderr parsl.dataflow.dflow INFO: Task 581 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d4059b5e63cc9c750eabb3c4aa917a97 with result from task 581 parsl.dataflow.dflow DEBUG: Task 581 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/da386ffd-8813-49aa-b365-39f2edc245b1_calibrate_855043_58.stdout parsl.dataflow.dflow INFO: Standard error for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/da386ffd-8813-49aa-b365-39f2edc245b1_calibrate_855043_58.stderr parsl.dataflow.dflow INFO: Task 490 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 334ce602ef81290d307724835b670792 with result from task 490 parsl.dataflow.dflow DEBUG: Task 490 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 491 has memoization hash 88b60c57d1345c554214e7b52493d818 parsl.dataflow.memoization INFO: Task 491 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456cd6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 491 try 0 launched on executor multi with executor id 927 parsl.dataflow.dflow INFO: Standard output for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/889a4534-172b-4a14-9126-f4db664621dc_calibrate_855044_25.stdout parsl.dataflow.dflow INFO: Standard error for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/889a4534-172b-4a14-9126-f4db664621dc_calibrate_855044_25.stderr parsl.dataflow.dflow INFO: Standard output for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/79228e1b-7b08-4fc6-8392-89d9e7424661_characterizeImage_855044_25.stdout parsl.dataflow.dflow INFO: Standard error for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/79228e1b-7b08-4fc6-8392-89d9e7424661_characterizeImage_855044_25.stderr parsl.dataflow.dflow INFO: Task 61 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9d7671408fb47acc568d597ae0de5a9 with result from task 61 parsl.dataflow.dflow DEBUG: Task 61 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 62 has memoization hash e71bc1cb7ed71638e048ff99085fc216 parsl.dataflow.memoization INFO: Task 62 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d2200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 62 try 0 launched on executor multi with executor id 928 parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/9e5189fa-751c-47f5-a214-52eaa99e8faa_calibrate_855138_60.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/9e5189fa-751c-47f5-a214-52eaa99e8faa_calibrate_855138_60.stderr parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/e184f266-fea7-41ec-995b-34cc43c5c41a_characterizeImage_855138_60.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/e184f266-fea7-41ec-995b-34cc43c5c41a_characterizeImage_855138_60.stderr parsl.dataflow.dflow INFO: Task 310 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69bae4c0ae94bab67f0f1396df58298e with result from task 310 parsl.dataflow.dflow DEBUG: Task 310 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 311 has memoization hash 6270507f00689182ea2f9bf1264ddf77 parsl.dataflow.memoization INFO: Task 311 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 311 try 0 launched on executor multi with executor id 929 parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/9dd0836d-1636-4357-a157-b9c4630a86f4_calibrate_855043_42.stdout parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/9dd0836d-1636-4357-a157-b9c4630a86f4_calibrate_855043_42.stderr parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/0f76515e-3707-43d8-afa2-41f8f832e03e_characterizeImage_855043_42.stdout parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/0f76515e-3707-43d8-afa2-41f8f832e03e_characterizeImage_855043_42.stderr parsl.dataflow.dflow INFO: Task 290 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13411f8f5b303f0f9151aea814c68bd4 with result from task 290 parsl.dataflow.dflow DEBUG: Task 290 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/ed44fa25-0bae-4993-b20e-3273d0b9632e_calibrate_855138_13.stdout parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/ed44fa25-0bae-4993-b20e-3273d0b9632e_calibrate_855138_13.stderr parsl.dataflow.dflow INFO: Task 424 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10e074e2aa0cf1704bc40417b732483a with result from task 424 parsl.dataflow.dflow DEBUG: Task 424 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 425 has memoization hash a13da714a6cff212721e31a993830818 parsl.dataflow.memoization INFO: Task 425 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a5b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 425 try 0 launched on executor multi with executor id 930 parsl.dataflow.dflow INFO: Standard output for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/d805cbfe-f750-43fb-8251-d515f2adf745_calibrate_855044_51.stdout parsl.dataflow.dflow INFO: Standard error for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/d805cbfe-f750-43fb-8251-d515f2adf745_calibrate_855044_51.stderr parsl.dataflow.dflow INFO: Standard output for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/d263fdd4-f7dc-414d-b52a-da9989834544_characterizeImage_855044_51.stdout parsl.dataflow.dflow INFO: Standard error for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/d263fdd4-f7dc-414d-b52a-da9989834544_characterizeImage_855044_51.stderr parsl.dataflow.dflow INFO: Task 277 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8681e04e2578377f621ab461b7458f4b with result from task 277 parsl.dataflow.dflow DEBUG: Task 277 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 278 has memoization hash 5f1804e83c06a2805b48057f40680184 parsl.dataflow.memoization INFO: Task 278 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 278 try 0 launched on executor multi with executor id 931 parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/4a32354a-fef0-4d69-9dd9-73854af53438_calibrate_855137_36.stdout parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/4a32354a-fef0-4d69-9dd9-73854af53438_calibrate_855137_36.stderr parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/f7352a57-b317-4e47-8790-85f807bfc37c_characterizeImage_855137_36.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/f7352a57-b317-4e47-8790-85f807bfc37c_characterizeImage_855137_36.stderr parsl.dataflow.dflow INFO: Task 841 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12527eef9962be6ba6ceeb982a921e1a with result from task 841 parsl.dataflow.dflow DEBUG: Task 841 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 842 has memoization hash c5455fb0ecef468fdc5d5dd29c344da1 parsl.dataflow.memoization INFO: Task 842 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45665260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 842 try 0 launched on executor multi with executor id 932 parsl.dataflow.dflow INFO: Standard output for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/19a8c8fb-74ea-4bd8-a7ef-30f637fc0451_calibrate_855137_42.stdout parsl.dataflow.dflow INFO: Standard error for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/19a8c8fb-74ea-4bd8-a7ef-30f637fc0451_calibrate_855137_42.stderr parsl.dataflow.dflow INFO: Standard output for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/b86aa0d6-36c8-4a4e-8071-87598f282e23_characterizeImage_855137_42.stdout parsl.dataflow.dflow INFO: Standard error for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/b86aa0d6-36c8-4a4e-8071-87598f282e23_characterizeImage_855137_42.stderr parsl.dataflow.dflow INFO: Task 701 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47b0e9abcddf32f5fc465860adfbc4b2 with result from task 701 parsl.dataflow.dflow DEBUG: Task 701 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/9ee55245-ff5b-41bf-b1a4-7066db455a01_calibrate_855043_45.stdout parsl.dataflow.dflow INFO: Standard error for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/9ee55245-ff5b-41bf-b1a4-7066db455a01_calibrate_855043_45.stderr parsl.dataflow.dflow INFO: Task 770 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 482ccc606cd1ffdbc5e9a98459c92c39 with result from task 770 parsl.dataflow.dflow DEBUG: Task 770 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/bc063c4f-063d-4481-a1e0-0212543d9629_calibrate_855137_1.stdout parsl.dataflow.dflow INFO: Standard error for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/bc063c4f-063d-4481-a1e0-0212543d9629_calibrate_855137_1.stderr parsl.dataflow.dflow INFO: Task 362 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e85bfe198afee346003b2c845710c423 with result from task 362 parsl.dataflow.dflow DEBUG: Task 362 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/9124c5d5-6eaf-4584-a53b-984a2175b4f2_calibrate_855137_25.stdout parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/9124c5d5-6eaf-4584-a53b-984a2175b4f2_calibrate_855137_25.stderr parsl.dataflow.dflow INFO: Task 961 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc3bed2fd567b0a4982ade5dbc6d3f09 with result from task 961 parsl.dataflow.dflow DEBUG: Task 961 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 962 has memoization hash 7f7301f9d693f83f0aaf595d79c4c00f parsl.dataflow.memoization INFO: Task 962 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454f8180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 962 try 0 launched on executor multi with executor id 933 parsl.dataflow.dflow INFO: Standard output for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/a7c8b557-db6b-4836-a5a0-ea61fb5992f8_calibrate_855043_27.stdout parsl.dataflow.dflow INFO: Standard error for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/a7c8b557-db6b-4836-a5a0-ea61fb5992f8_calibrate_855043_27.stderr parsl.dataflow.dflow INFO: Standard output for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/e4d6d4c2-63dd-4dda-95d6-476aa224b354_characterizeImage_855043_27.stdout parsl.dataflow.dflow INFO: Standard error for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/e4d6d4c2-63dd-4dda-95d6-476aa224b354_characterizeImage_855043_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:18:52 proc_lsst.multi:146 [INFO] found job 16912300 in provider astro proc_lsst.multi INFO: found job 16912300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 20 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c5b9d5b74e90f54b2dd52473c4dd3fd with result from task 20 parsl.dataflow.dflow DEBUG: Task 20 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/f9c8778d-75cd-4aa9-a668-a3562a6fa456_calibrate_855044_3.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/f9c8778d-75cd-4aa9-a668-a3562a6fa456_calibrate_855044_3.stderr parsl.dataflow.dflow INFO: Task 41 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be46eaad2ae3625fab2cb12d24746b5e with result from task 41 parsl.dataflow.dflow DEBUG: Task 41 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/9a4a1baa-c422-450a-9a4b-a9a99c30b564_calibrate_855042_62.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/9a4a1baa-c422-450a-9a4b-a9a99c30b564_calibrate_855042_62.stderr parsl.dataflow.dflow INFO: Task 664 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a27fb6b673dde82ec26048609a1892eb with result from task 664 parsl.dataflow.dflow DEBUG: Task 664 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 665 has memoization hash d79729f9328b99e1dde13b790910abd5 parsl.dataflow.memoization INFO: Task 665 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 665 try 0 launched on executor multi with executor id 934 parsl.dataflow.dflow INFO: Standard output for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/670d6def-4de8-4de7-b085-b2fd8758aa18_calibrate_855042_18.stdout parsl.dataflow.dflow INFO: Standard error for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/670d6def-4de8-4de7-b085-b2fd8758aa18_calibrate_855042_18.stderr parsl.dataflow.dflow INFO: Standard output for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/59653cd9-1bdd-412d-a9b9-422a8195672a_characterizeImage_855042_18.stdout parsl.dataflow.dflow INFO: Standard error for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/59653cd9-1bdd-412d-a9b9-422a8195672a_characterizeImage_855042_18.stderr parsl.dataflow.dflow INFO: Task 910 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d95f1f31a56b8c0f8b8a2007d25d0948 with result from task 910 parsl.dataflow.dflow DEBUG: Task 910 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 911 has memoization hash 4e392534cf53919d702d4fc88a3b66a7 parsl.dataflow.memoization INFO: Task 911 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45499da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 911 try 0 launched on executor multi with executor id 935 parsl.dataflow.dflow INFO: Standard output for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/d9937d5d-a352-4c6f-87ce-3aece9e63183_calibrate_855043_8.stdout parsl.dataflow.dflow INFO: Standard error for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/d9937d5d-a352-4c6f-87ce-3aece9e63183_calibrate_855043_8.stderr parsl.dataflow.dflow INFO: Standard output for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/e8be66b1-62e1-42c2-9ee1-fef43c057cba_characterizeImage_855043_8.stdout parsl.dataflow.dflow INFO: Standard error for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/e8be66b1-62e1-42c2-9ee1-fef43c057cba_characterizeImage_855043_8.stderr parsl.dataflow.dflow INFO: Task 931 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4825bdc9fa9e2fc1e5cbc57c93c8def with result from task 931 parsl.dataflow.dflow DEBUG: Task 931 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 932 has memoization hash caa4443c2e938dbfbf700f10a8cfb86b parsl.dataflow.memoization INFO: Task 932 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4549b1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 932 try 0 launched on executor multi with executor id 936 parsl.dataflow.dflow INFO: Standard output for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/1b6b1ab7-f2c5-4ded-b5ce-ba89f56e4cf9_calibrate_855136_55.stdout parsl.dataflow.dflow INFO: Standard error for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/1b6b1ab7-f2c5-4ded-b5ce-ba89f56e4cf9_calibrate_855136_55.stderr parsl.dataflow.dflow INFO: Standard output for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/b2eed188-6b29-4e8d-9d75-c0b6dd463507_characterizeImage_855136_55.stdout parsl.dataflow.dflow INFO: Standard error for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/b2eed188-6b29-4e8d-9d75-c0b6dd463507_characterizeImage_855136_55.stderr parsl.dataflow.dflow INFO: Task 710 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8f25b24782df584460d0f0052cb30f5 with result from task 710 parsl.dataflow.dflow DEBUG: Task 710 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/32499465-8270-4f57-b787-2511fa37d6f3_calibrate_855042_5.stdout parsl.dataflow.dflow INFO: Standard error for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/32499465-8270-4f57-b787-2511fa37d6f3_calibrate_855042_5.stderr parsl.dataflow.dflow INFO: Task 484 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a7a9f5e5498c0c60505fad0c6960a1a1 with result from task 484 parsl.dataflow.dflow DEBUG: Task 484 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 485 has memoization hash 258f33fc2fe25dcaf3e4f2b691a5c9c2 parsl.dataflow.memoization INFO: Task 485 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456ccfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 485 try 0 launched on executor multi with executor id 937 parsl.dataflow.dflow INFO: Standard output for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/1950b431-c469-4808-8043-19acbd8b7ada_calibrate_855136_27.stdout parsl.dataflow.dflow INFO: Standard error for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/1950b431-c469-4808-8043-19acbd8b7ada_calibrate_855136_27.stderr parsl.dataflow.dflow INFO: Standard output for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/7269bfa7-181f-45aa-a990-1a46870a9334_characterizeImage_855136_27.stdout parsl.dataflow.dflow INFO: Standard error for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/7269bfa7-181f-45aa-a990-1a46870a9334_characterizeImage_855136_27.stderr parsl.dataflow.dflow INFO: Task 967 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7cac6dabdefa7c212efc50a0d445d397 with result from task 967 parsl.dataflow.dflow DEBUG: Task 967 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 968 has memoization hash ce1eaf46384e0c1f0698f5c17850297a parsl.dataflow.memoization INFO: Task 968 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454f8860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 968 try 0 launched on executor multi with executor id 938 parsl.dataflow.dflow INFO: Standard output for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/5a96b929-1001-47a8-91f6-835a2723ac82_calibrate_855043_7.stdout parsl.dataflow.dflow INFO: Standard error for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/5a96b929-1001-47a8-91f6-835a2723ac82_calibrate_855043_7.stderr parsl.dataflow.dflow INFO: Standard output for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/2d68c8c5-937c-45a1-9ea5-03f002e0a657_characterizeImage_855043_7.stdout parsl.dataflow.dflow INFO: Standard error for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/2d68c8c5-937c-45a1-9ea5-03f002e0a657_characterizeImage_855043_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:18:54 proc_lsst.multi:146 [INFO] found job 16912301 in provider astro proc_lsst.multi INFO: found job 16912301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 958 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d29d89c96f3844e9df48fc39e41da473 with result from task 958 parsl.dataflow.dflow DEBUG: Task 958 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 959 has memoization hash 2a075323ff66790f17be297a9a490de9 parsl.dataflow.memoization INFO: Task 959 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454fa340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 959 try 0 launched on executor multi with executor id 939 parsl.dataflow.dflow INFO: Standard output for task 959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/b9a96252-f87d-4ab2-836e-75b73ff408bb_calibrate_855137_22.stdout parsl.dataflow.dflow INFO: Standard error for task 959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/b9a96252-f87d-4ab2-836e-75b73ff408bb_calibrate_855137_22.stderr parsl.dataflow.dflow INFO: Standard output for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/464e5e09-2ad3-41b6-896c-c8e625e2f148_characterizeImage_855137_22.stdout parsl.dataflow.dflow INFO: Standard error for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/464e5e09-2ad3-41b6-896c-c8e625e2f148_characterizeImage_855137_22.stderr parsl.dataflow.dflow INFO: Task 646 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bfe2266b5d9ce3fc4f0d27379d2d8574 with result from task 646 parsl.dataflow.dflow DEBUG: Task 646 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 647 has memoization hash 007228ffd4136869e6ffd0906871ebbe parsl.dataflow.memoization INFO: Task 647 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 647 try 0 launched on executor multi with executor id 940 parsl.dataflow.dflow INFO: Standard output for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/5f59fbdb-5863-4074-826c-779094b522f8_calibrate_855043_44.stdout parsl.dataflow.dflow INFO: Standard error for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/5f59fbdb-5863-4074-826c-779094b522f8_calibrate_855043_44.stderr parsl.dataflow.dflow INFO: Standard output for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/dd732501-883e-4c66-952e-f5ffa4d0ee73_characterizeImage_855043_44.stdout parsl.dataflow.dflow INFO: Standard error for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/dd732501-883e-4c66-952e-f5ffa4d0ee73_characterizeImage_855043_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:18:55 proc_lsst.multi:146 [INFO] found job 16912302 in provider astro proc_lsst.multi INFO: found job 16912302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 358 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 495f43acad4696df7be22976dc4efac8 with result from task 358 parsl.dataflow.dflow DEBUG: Task 358 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 359 has memoization hash f0ae943ceddb23ad673216464ee80c58 parsl.dataflow.memoization INFO: Task 359 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa468798a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 359 try 0 launched on executor multi with executor id 941 parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/0e80ede9-c775-4be3-b2fc-bb9609365bd4_calibrate_855042_34.stdout parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/0e80ede9-c775-4be3-b2fc-bb9609365bd4_calibrate_855042_34.stderr parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/29b429dc-034d-4303-9d84-2e4dc6ee64e7_characterizeImage_855042_34.stdout parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/29b429dc-034d-4303-9d84-2e4dc6ee64e7_characterizeImage_855042_34.stderr parsl.dataflow.dflow INFO: Task 776 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 558263fe67fae5a535b0ba23e584c1e4 with result from task 776 parsl.dataflow.dflow DEBUG: Task 776 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/879c90aa-9458-48c9-8ad7-205ea964d056_calibrate_855138_38.stdout parsl.dataflow.dflow INFO: Standard error for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/879c90aa-9458-48c9-8ad7-205ea964d056_calibrate_855138_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:18:56 proc_lsst.multi:146 [INFO] found job 16912303 in provider astro proc_lsst.multi INFO: found job 16912303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 208 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ecad099e0db6b7cd46973c9a6ee4424 with result from task 208 parsl.dataflow.dflow DEBUG: Task 208 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 209 has memoization hash 8c03cf38e53517d318be14161c3a0254 parsl.dataflow.memoization INFO: Task 209 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46817ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 209 try 0 launched on executor multi with executor id 942 parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/e4671f0f-5fc2-412b-afa2-61ab06b56ee2_calibrate_855042_17.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/e4671f0f-5fc2-412b-afa2-61ab06b56ee2_calibrate_855042_17.stderr parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/fb0a4c66-200e-4adf-a6b8-30eb7e648b09_characterizeImage_855042_17.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/fb0a4c66-200e-4adf-a6b8-30eb7e648b09_characterizeImage_855042_17.stderr parsl.dataflow.dflow INFO: Task 202 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 265306c6720391e7dca37cbf6dd42491 with result from task 202 parsl.dataflow.dflow DEBUG: Task 202 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 203 has memoization hash ebe17a92e0f52eb467ca3ce90fea5a2e parsl.dataflow.memoization INFO: Task 203 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46817560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 203 try 0 launched on executor multi with executor id 943 parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/eef30ba1-32e2-4fde-a71e-7d6c53a6c420_calibrate_855138_33.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/eef30ba1-32e2-4fde-a71e-7d6c53a6c420_calibrate_855138_33.stderr parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/df792072-9c16-47c5-8de8-896e77edbed2_characterizeImage_855138_33.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/df792072-9c16-47c5-8de8-896e77edbed2_characterizeImage_855138_33.stderr parsl.dataflow.dflow INFO: Task 262 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 103b49ba7bbdec178dea05e5772d6734 with result from task 262 parsl.dataflow.dflow DEBUG: Task 262 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 263 has memoization hash b0e7bcd43e58a9b11ed685059b93dd3c parsl.dataflow.memoization INFO: Task 263 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46837420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 263 try 0 launched on executor multi with executor id 944 parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/eebcc2a4-e110-45e5-a603-867bf34db737_calibrate_855136_21.stdout parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/eebcc2a4-e110-45e5-a603-867bf34db737_calibrate_855136_21.stderr parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/708151c8-7239-4854-bb42-5de7caa258ee_characterizeImage_855136_21.stdout parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/708151c8-7239-4854-bb42-5de7caa258ee_characterizeImage_855136_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:18:57 proc_lsst.multi:146 [INFO] found job 16912304 in provider astro proc_lsst.multi INFO: found job 16912304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 172 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e90c6592c3c15e1ab6eb217f5aa1fd06 with result from task 172 parsl.dataflow.dflow DEBUG: Task 172 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 173 has memoization hash 0433bc892be1f46d75b7f3c759408521 parsl.dataflow.memoization INFO: Task 173 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46815620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 173 try 0 launched on executor multi with executor id 945 parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/e6020d81-39fb-4fc7-a747-51346446508a_calibrate_855138_49.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/e6020d81-39fb-4fc7-a747-51346446508a_calibrate_855138_49.stderr parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/96c60d0d-3417-4a6c-9754-6a5095204de4_characterizeImage_855138_49.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/96c60d0d-3417-4a6c-9754-6a5095204de4_characterizeImage_855138_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:18:57 proc_lsst.multi:146 [INFO] found job 16912305 in provider astro proc_lsst.multi INFO: found job 16912305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 761 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d759ab6550c6f8b2ec1fee021e34022 with result from task 761 parsl.dataflow.dflow DEBUG: Task 761 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/309b3cd3-866a-43fa-aa5c-eca8039b986e_calibrate_855043_21.stdout parsl.dataflow.dflow INFO: Standard error for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/309b3cd3-866a-43fa-aa5c-eca8039b986e_calibrate_855043_21.stderr parsl.dataflow.dflow INFO: Task 746 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 169223691d1dbe4e7ab36ce5e2897185 with result from task 746 parsl.dataflow.dflow DEBUG: Task 746 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/fe932c5e-156a-4c2d-97fc-7485398d23cc_calibrate_855043_15.stdout parsl.dataflow.dflow INFO: Standard error for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/fe932c5e-156a-4c2d-97fc-7485398d23cc_calibrate_855043_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:18:58 proc_lsst.multi:146 [INFO] found job 16912306 in provider astro proc_lsst.multi INFO: found job 16912306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 256 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a8082e57c9fb477d54c293cfa3d83c6 with result from task 256 parsl.dataflow.dflow DEBUG: Task 256 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 257 has memoization hash a2e098e2bb529a62016b58b68bc38b17 parsl.dataflow.memoization INFO: Task 257 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46836de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 257 try 0 launched on executor multi with executor id 946 parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/a6621236-1fa4-4a66-bd98-7c70f48c36f7_calibrate_855042_43.stdout parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/a6621236-1fa4-4a66-bd98-7c70f48c36f7_calibrate_855042_43.stderr parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/33c9fa3e-f8ea-45a4-820b-42de9469e0a3_characterizeImage_855042_43.stdout parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/33c9fa3e-f8ea-45a4-820b-42de9469e0a3_characterizeImage_855042_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:18:59 proc_lsst.multi:146 [INFO] found job 16912307 in provider astro proc_lsst.multi INFO: found job 16912307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 508 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6fe65ebd1b5b20ea9833c939cc1e2319 with result from task 508 parsl.dataflow.dflow DEBUG: Task 508 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 509 has memoization hash bf5175b53a6565d4408858f75da61bb3 parsl.dataflow.memoization INFO: Task 509 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456cea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 509 try 0 launched on executor multi with executor id 947 parsl.dataflow.dflow INFO: Standard output for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/115b8f32-d876-48e5-9ff6-6a05bdd7f56a_calibrate_855044_27.stdout parsl.dataflow.dflow INFO: Standard error for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/115b8f32-d876-48e5-9ff6-6a05bdd7f56a_calibrate_855044_27.stderr parsl.dataflow.dflow INFO: Standard output for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/a8bbf969-b430-43a0-80b5-b979329aede0_characterizeImage_855044_27.stdout parsl.dataflow.dflow INFO: Standard error for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/a8bbf969-b430-43a0-80b5-b979329aede0_characterizeImage_855044_27.stderr parsl.dataflow.dflow INFO: Task 319 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 361e0734c952c21a88272390a1c120cf with result from task 319 parsl.dataflow.dflow DEBUG: Task 319 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 320 has memoization hash 3d9df48ea8ec1946b26b06e86cc20e2c parsl.dataflow.memoization INFO: Task 320 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 320 try 0 launched on executor multi with executor id 948 parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/ff183dc0-60ac-4295-a81f-ab7573aacc95_calibrate_855044_56.stdout parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/ff183dc0-60ac-4295-a81f-ab7573aacc95_calibrate_855044_56.stderr parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/ea8da8c3-49f3-48cc-a161-0fbd536d8a82_characterizeImage_855044_56.stdout parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/ea8da8c3-49f3-48cc-a161-0fbd536d8a82_characterizeImage_855044_56.stderr parsl.dataflow.dflow INFO: Task 131 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0127ca55dc8652c1c3020e148bdb7bdc with result from task 131 parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:00 proc_lsst.multi:146 [INFO] found job 16912308 in provider astro proc_lsst.multi INFO: found job 16912308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow DEBUG: Task 131 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/27af0d40-1608-43c3-b008-702315fc5bb3_calibrate_855042_3.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/27af0d40-1608-43c3-b008-702315fc5bb3_calibrate_855042_3.stderr parsl.dataflow.dflow INFO: Task 328 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e25904ffa816835242d3f9d89f3bb963 with result from task 328 parsl.dataflow.dflow DEBUG: Task 328 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 329 has memoization hash 45a674d885c5ce9c752893da181ec75a parsl.dataflow.memoization INFO: Task 329 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 329 try 0 launched on executor multi with executor id 949 parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/839f676e-715a-4d53-989c-57288b0ecede_calibrate_855138_58.stdout parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/839f676e-715a-4d53-989c-57288b0ecede_calibrate_855138_58.stderr parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/44bed20b-ca1d-4cbd-aa34-7d2c2a5f3e22_characterizeImage_855138_58.stdout parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/44bed20b-ca1d-4cbd-aa34-7d2c2a5f3e22_characterizeImage_855138_58.stderr parsl.dataflow.dflow INFO: Task 724 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dbce7b778b3d9096ce9635dbc9e157d4 with result from task 724 parsl.dataflow.dflow DEBUG: Task 724 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 725 has memoization hash 4e67f4e4fa3fe4be88db42d9520a5dc0 parsl.dataflow.memoization INFO: Task 725 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d1940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 725 try 0 launched on executor multi with executor id 950 parsl.dataflow.dflow INFO: Standard output for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/628228aa-6b1f-4426-85a1-6b6bfcf806a9_calibrate_855138_12.stdout parsl.dataflow.dflow INFO: Standard error for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/628228aa-6b1f-4426-85a1-6b6bfcf806a9_calibrate_855138_12.stderr parsl.dataflow.dflow INFO: Standard output for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/c5067539-eb35-47f6-bd36-f9909c37b581_characterizeImage_855138_12.stdout parsl.dataflow.dflow INFO: Standard error for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/c5067539-eb35-47f6-bd36-f9909c37b581_characterizeImage_855138_12.stderr parsl.dataflow.dflow INFO: Task 949 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5cbf1bee8d85e492d5cdf2162e2408bf with result from task 949 parsl.dataflow.dflow DEBUG: Task 949 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 950 has memoization hash c2ae92a5e8d4e0854e259668a19c96ab parsl.dataflow.memoization INFO: Task 950 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454fa980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 950 try 0 launched on executor multi with executor id 951 parsl.dataflow.dflow INFO: Standard output for task 950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/451b17f0-ad0c-418f-a833-f10d0035d841_calibrate_855136_58.stdout parsl.dataflow.dflow INFO: Standard error for task 950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/451b17f0-ad0c-418f-a833-f10d0035d841_calibrate_855136_58.stderr parsl.dataflow.dflow INFO: Standard output for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/07263472-46b8-4b80-8985-a0d1bde7c412_characterizeImage_855136_58.stdout parsl.dataflow.dflow INFO: Standard error for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/07263472-46b8-4b80-8985-a0d1bde7c412_characterizeImage_855136_58.stderr parsl.dataflow.dflow INFO: Task 98 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 45e8ed9b1f1296e9655dac000df0cbc5 with result from task 98 parsl.dataflow.dflow DEBUG: Task 98 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/c2018014-5353-4b60-903a-e8153c50efb3_calibrate_855042_55.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/c2018014-5353-4b60-903a-e8153c50efb3_calibrate_855042_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:00 proc_lsst.multi:146 [INFO] found job 16912309 in provider astro proc_lsst.multi INFO: found job 16912309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 773 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 41f428cd545b3635e8ce3a2d10b541c7 with result from task 773 parsl.dataflow.dflow DEBUG: Task 773 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/c1f7ecec-0123-497a-a8ce-01e2bceacb1a_calibrate_855043_37.stdout parsl.dataflow.dflow INFO: Standard error for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/c1f7ecec-0123-497a-a8ce-01e2bceacb1a_calibrate_855043_37.stderr parsl.dataflow.dflow INFO: Task 229 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3e917f7aa5058ae823b69bc6de1101c with result from task 229 parsl.dataflow.dflow DEBUG: Task 229 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 230 has memoization hash 73a47c5805e1f7190a0467ef0822445b parsl.dataflow.memoization INFO: Task 230 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa468351c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 230 try 0 launched on executor multi with executor id 952 parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/08f565e1-117d-40d9-a9d9-ccf131555b50_calibrate_855137_32.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/08f565e1-117d-40d9-a9d9-ccf131555b50_calibrate_855137_32.stderr parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/dff41e58-7500-46f3-b543-95f755274de4_characterizeImage_855137_32.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/dff41e58-7500-46f3-b543-95f755274de4_characterizeImage_855137_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:01 proc_lsst.multi:146 [INFO] found job 16912310 in provider astro proc_lsst.multi INFO: found job 16912310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 655 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 43a39929fec5d01bd6422e4bb576bfd6 with result from task 655 parsl.dataflow.dflow DEBUG: Task 655 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 656 has memoization hash 5346a4fd026de12a3bffebbb9a3dea22 parsl.dataflow.memoization INFO: Task 656 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 656 try 0 launched on executor multi with executor id 953 parsl.dataflow.dflow INFO: Standard output for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/ca50a906-23ee-46dc-847a-67558d460985_calibrate_855043_39.stdout parsl.dataflow.dflow INFO: Standard error for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/ca50a906-23ee-46dc-847a-67558d460985_calibrate_855043_39.stderr parsl.dataflow.dflow INFO: Standard output for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/5b3e887d-5921-44e4-802b-329643e5dc09_characterizeImage_855043_39.stdout parsl.dataflow.dflow INFO: Standard error for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/5b3e887d-5921-44e4-802b-329643e5dc09_characterizeImage_855043_39.stderr parsl.dataflow.dflow INFO: Task 964 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1085ca6a8afd1995f01483c137c4e0f9 with result from task 964 parsl.dataflow.dflow DEBUG: Task 964 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 965 has memoization hash b7f9eed2bac533fcaf88616d40655601 parsl.dataflow.memoization INFO: Task 965 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454f8540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 965 try 0 launched on executor multi with executor id 954 parsl.dataflow.dflow INFO: Standard output for task 965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/237cfd05-6de1-446d-9ef7-fa521434be1c_calibrate_855043_54.stdout parsl.dataflow.dflow INFO: Standard error for task 965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/237cfd05-6de1-446d-9ef7-fa521434be1c_calibrate_855043_54.stderr parsl.dataflow.dflow INFO: Standard output for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/66e0d236-d259-4777-86af-60f4dadf04fe_characterizeImage_855043_54.stdout parsl.dataflow.dflow INFO: Standard error for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/66e0d236-d259-4777-86af-60f4dadf04fe_characterizeImage_855043_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:02 proc_lsst.multi:146 [INFO] found job 16912311 in provider astro proc_lsst.multi INFO: found job 16912311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:03 proc_lsst.multi:146 [INFO] found job 16912312 in provider astro proc_lsst.multi INFO: found job 16912312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 988 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24bba4fdeb7378f98cb7c2daf6aacfdb with result from task 988 parsl.dataflow.dflow DEBUG: Task 988 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 989 has memoization hash 373315e3e94c15ad3788fc73ef948c70 parsl.dataflow.memoization INFO: Task 989 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454fa020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 989 try 0 launched on executor multi with executor id 955 parsl.dataflow.dflow INFO: Standard output for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/c1979fb4-f681-4221-9fdb-ab08ca5412e2_calibrate_855136_51.stdout parsl.dataflow.dflow INFO: Standard error for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/c1979fb4-f681-4221-9fdb-ab08ca5412e2_calibrate_855136_51.stderr parsl.dataflow.dflow INFO: Standard output for task 988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/09fbcbaa-9568-4ab1-8dfa-7ceb1d2b3af1_characterizeImage_855136_51.stdout parsl.dataflow.dflow INFO: Standard error for task 988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/09fbcbaa-9568-4ab1-8dfa-7ceb1d2b3af1_characterizeImage_855136_51.stderr parsl.dataflow.dflow INFO: Task 847 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3f82bb23a084bc1728bd70d66b14710 with result from task 847 parsl.dataflow.dflow DEBUG: Task 847 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 848 has memoization hash 19b5ae5a837423b362f7dc35c78e2e39 parsl.dataflow.memoization INFO: Task 848 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456659e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 848 try 0 launched on executor multi with executor id 956 parsl.dataflow.dflow INFO: Standard output for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/3cc80c49-d469-410b-aca4-de6e72971a0c_calibrate_855044_7.stdout parsl.dataflow.dflow INFO: Standard error for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/3cc80c49-d469-410b-aca4-de6e72971a0c_calibrate_855044_7.stderr parsl.dataflow.dflow INFO: Standard output for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/c4095486-07d4-4eb1-8529-5349f41b3a3e_characterizeImage_855044_7.stdout parsl.dataflow.dflow INFO: Standard error for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/c4095486-07d4-4eb1-8529-5349f41b3a3e_characterizeImage_855044_7.stderr parsl.dataflow.dflow INFO: Task 376 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry faf13cbb836b7d198a98e93b8ad37341 with result from task 376 parsl.dataflow.dflow DEBUG: Task 376 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 377 has memoization hash 6bbd114296741b676ec6533a2bf426cd parsl.dataflow.memoization INFO: Task 377 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a5ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 377 try 0 launched on executor multi with executor id 957 parsl.dataflow.dflow INFO: Standard output for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/babbdd25-c78e-491d-be3d-7894f9629f68_calibrate_855042_29.stdout parsl.dataflow.dflow INFO: Standard error for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/babbdd25-c78e-491d-be3d-7894f9629f68_calibrate_855042_29.stderr parsl.dataflow.dflow INFO: Standard output for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/4e4de332-b56d-47b4-ab60-a131819b3bfd_characterizeImage_855042_29.stdout parsl.dataflow.dflow INFO: Standard error for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/4e4de332-b56d-47b4-ab60-a131819b3bfd_characterizeImage_855042_29.stderr parsl.dataflow.dflow INFO: Task 743 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2f700d10a06e7d3636f708002e00b11 with result from task 743 parsl.dataflow.dflow DEBUG: Task 743 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/de6beae1-c85c-44db-98f4-3ac0e345c834_calibrate_855042_50.stdout parsl.dataflow.dflow INFO: Standard error for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/de6beae1-c85c-44db-98f4-3ac0e345c834_calibrate_855042_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:03 proc_lsst.multi:146 [INFO] found job 16912313 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912313 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.dataflow.dflow INFO: Task 323 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 215e2f19ab8d82fa0bcd1799c1a5f5ac with result from task 323 parsl.dataflow.dflow DEBUG: Task 323 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/dd174b2e-e289-4f22-acd6-73ee59553a04_calibrate_855042_28.stdout parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/dd174b2e-e289-4f22-acd6-73ee59553a04_calibrate_855042_28.stderr parsl.dataflow.dflow INFO: Task 886 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d47683908acf87f75a95b9cee390ead with result from task 886 parsl.dataflow.dflow DEBUG: Task 886 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 887 has memoization hash 0b49bd462b5832d9b3feee00fdf18b21 parsl.dataflow.memoization INFO: Task 887 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45667ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 887 try 0 launched on executor multi with executor id 958 parsl.dataflow.dflow INFO: Standard output for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/18165b7f-aff8-4257-9ab7-4c8cd15da16e_calibrate_855042_53.stdout parsl.dataflow.dflow INFO: Standard error for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/18165b7f-aff8-4257-9ab7-4c8cd15da16e_calibrate_855042_53.stderr parsl.dataflow.dflow INFO: Standard output for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/993efa91-b6f2-40b5-9730-6867d85689c3_characterizeImage_855042_53.stdout parsl.dataflow.dflow INFO: Standard error for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/993efa91-b6f2-40b5-9730-6867d85689c3_characterizeImage_855042_53.stderr parsl.dataflow.dflow INFO: Task 38 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c04088ad2243d7d54957386e1d4bf2e6 with result from task 38 parsl.dataflow.dflow DEBUG: Task 38 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/0dab76d9-b659-491b-922d-08f0dc77e48a_calibrate_855138_42.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/0dab76d9-b659-491b-922d-08f0dc77e48a_calibrate_855138_42.stderr parsl.dataflow.dflow INFO: Task 791 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a331f2827c1e2de40c250019f2f1b90 with result from task 791 parsl.dataflow.dflow DEBUG: Task 791 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/daed5e80-2a0b-4e5d-a5ef-8972ba41dd3a_calibrate_855044_35.stdout parsl.dataflow.dflow INFO: Standard error for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/daed5e80-2a0b-4e5d-a5ef-8972ba41dd3a_calibrate_855044_35.stderr parsl.dataflow.dflow INFO: Task 925 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 791fd6e6b58e64604632cfff47c9089e with result from task 925 parsl.dataflow.dflow DEBUG: Task 925 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 926 has memoization hash b888f4c9bbe884b2e598f366dd516641 parsl.dataflow.memoization INFO: Task 926 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4549ab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 926 try 0 launched on executor multi with executor id 959 parsl.dataflow.dflow INFO: Standard output for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/d5149127-61cc-4977-b9bb-f101b5823d14_calibrate_855138_46.stdout parsl.dataflow.dflow INFO: Standard error for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/d5149127-61cc-4977-b9bb-f101b5823d14_calibrate_855138_46.stderr parsl.dataflow.dflow INFO: Standard output for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/eff08db8-a532-4f51-b9ce-bd03d526f6bc_characterizeImage_855138_46.stdout parsl.dataflow.dflow INFO: Standard error for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/eff08db8-a532-4f51-b9ce-bd03d526f6bc_characterizeImage_855138_46.stderr parsl.dataflow.dflow INFO: Task 199 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1cee8ddcefc1cc1530b2729abe60d3ba with result from task 199 parsl.dataflow.dflow DEBUG: Task 199 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 200 has memoization hash 6f40c6645e075c2756b87d71feb91ccf parsl.dataflow.memoization INFO: Task 200 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46817240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 200 try 0 launched on executor multi with executor id 960 parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/512b77a2-c861-46a7-839b-49ade36d2b2f_calibrate_855042_10.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/512b77a2-c861-46a7-839b-49ade36d2b2f_calibrate_855042_10.stderr parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/46a95e67-16a4-41aa-9016-9e0294305b05_characterizeImage_855042_10.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/46a95e67-16a4-41aa-9016-9e0294305b05_characterizeImage_855042_10.stderr parsl.dataflow.dflow INFO: Task 367 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6dd50be2840f3a7051b2b03b582b0010 with result from task 367 parsl.dataflow.dflow DEBUG: Task 367 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 368 has memoization hash 539a109af666bf331021a5b4c658656c parsl.dataflow.memoization INFO: Task 368 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4687a200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 368 try 0 launched on executor multi with executor id 961 parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/8a8ce4f8-7d92-4135-85eb-9cd5ebc26ff7_calibrate_855044_17.stdout parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/8a8ce4f8-7d92-4135-85eb-9cd5ebc26ff7_calibrate_855044_17.stderr parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/d32e73d8-8360-4b1a-9b1d-888889c9057c_characterizeImage_855044_17.stdout parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/d32e73d8-8360-4b1a-9b1d-888889c9057c_characterizeImage_855044_17.stderr parsl.dataflow.dflow INFO: Task 235 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a6b9bae6e3268fe106dd516a45b7739 with result from task 235 parsl.dataflow.dflow DEBUG: Task 235 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 236 has memoization hash a9ac7fc5591f25ad4ad9b643cb32cf25 parsl.dataflow.memoization INFO: Task 236 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46835800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 236 try 0 launched on executor multi with executor id 962 parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/4aa15a4c-fd0d-46cd-aef5-c136cfc61e99_calibrate_855044_16.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/4aa15a4c-fd0d-46cd-aef5-c136cfc61e99_calibrate_855044_16.stderr parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/4d69cde3-bc32-4836-86e1-34834fb92a43_characterizeImage_855044_16.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/4d69cde3-bc32-4836-86e1-34834fb92a43_characterizeImage_855044_16.stderr parsl.dataflow.dflow INFO: Task 122 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ded91c991ac29cb585d792e5f92faf49 with result from task 122 parsl.dataflow.dflow DEBUG: Task 122 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/7d33d7f9-651e-4e93-a8c3-7efd5e25d2b7_calibrate_855044_57.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/7d33d7f9-651e-4e93-a8c3-7efd5e25d2b7_calibrate_855044_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:04 proc_lsst.multi:146 [INFO] found job 16912314 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912314 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.dataflow.dflow INFO: Task 643 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ffb841b5f269d82827892b35c99b7b6c with result from task 643 parsl.dataflow.dflow DEBUG: Task 643 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 644 has memoization hash 53d9738db3183bf827e05916062f133b parsl.dataflow.memoization INFO: Task 644 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 644 try 0 launched on executor multi with executor id 963 parsl.dataflow.dflow INFO: Standard output for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/f1f98230-728f-4d9e-b00b-f7c48b76b32e_calibrate_855043_33.stdout parsl.dataflow.dflow INFO: Standard error for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/f1f98230-728f-4d9e-b00b-f7c48b76b32e_calibrate_855043_33.stderr parsl.dataflow.dflow INFO: Standard output for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/05450563-8e9d-4013-8aa9-dd2e2e248f33_characterizeImage_855043_33.stdout parsl.dataflow.dflow INFO: Standard error for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/05450563-8e9d-4013-8aa9-dd2e2e248f33_characterizeImage_855043_33.stderr parsl.dataflow.dflow INFO: Task 649 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a6fbf30db7d8d084ed87f82579f43b0f with result from task 649 parsl.dataflow.dflow DEBUG: Task 649 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 650 has memoization hash 3ae6180eefdda8c02843bb86321c55b3 parsl.dataflow.memoization INFO: Task 650 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 650 try 0 launched on executor multi with executor id 964 parsl.dataflow.dflow INFO: Standard output for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/b6e2b540-68e4-4956-a685-fae8fb13fa12_calibrate_855137_26.stdout parsl.dataflow.dflow INFO: Standard error for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/b6e2b540-68e4-4956-a685-fae8fb13fa12_calibrate_855137_26.stderr parsl.dataflow.dflow INFO: Standard output for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/670e7359-a626-40a5-b51f-69d14a21fb27_characterizeImage_855137_26.stdout parsl.dataflow.dflow INFO: Standard error for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/670e7359-a626-40a5-b51f-69d14a21fb27_characterizeImage_855137_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:05 proc_lsst.multi:146 [INFO] found job 16912315 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912315 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.dataflow.dflow INFO: Task 386 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d6ae31ed421517f5e87efdbcb4fcbec with result from task 386 parsl.dataflow.dflow DEBUG: Task 386 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/d91c5031-e1ab-4162-af09-8af059d0fcaf_calibrate_855136_1.stdout parsl.dataflow.dflow INFO: Standard error for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/d91c5031-e1ab-4162-af09-8af059d0fcaf_calibrate_855136_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:06 proc_lsst.multi:146 [INFO] found job 16912316 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912316 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.dataflow.dflow INFO: Task 1000 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b47d24bc47b2c4681447e359c6436fb with result from task 1000 parsl.dataflow.dflow DEBUG: Task 1000 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1001 has memoization hash b505e34c10e773072bfe4c0379e1f995 parsl.dataflow.memoization INFO: Task 1001 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454fbba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1001 try 0 launched on executor multi with executor id 965 parsl.dataflow.dflow INFO: Standard output for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/b86ea807-bd1e-4af6-83b8-f35d441405bd_calibrate_855044_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/b86ea807-bd1e-4af6-83b8-f35d441405bd_calibrate_855044_49.stderr parsl.dataflow.dflow INFO: Standard output for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/a96ef3dd-745b-4858-85cd-d464c276b70f_characterizeImage_855044_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/a96ef3dd-745b-4858-85cd-d464c276b70f_characterizeImage_855044_49.stderr parsl.dataflow.dflow INFO: Task 71 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ffc6212f1ec24e34ea37cda4676e444 with result from task 71 parsl.dataflow.dflow DEBUG: Task 71 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/ee1c9923-93d0-4878-96bc-d8aaa67f64f5_calibrate_855137_4.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/ee1c9923-93d0-4878-96bc-d8aaa67f64f5_calibrate_855137_4.stderr parsl.dataflow.dflow INFO: Task 637 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f30f6235571db374d6a0fff9a103cbcb with result from task 637 parsl.dataflow.dflow DEBUG: Task 637 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 638 has memoization hash 3e87edd3f163b21e01289bd7dcb37e0d parsl.dataflow.memoization INFO: Task 638 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 638 try 0 launched on executor multi with executor id 966 parsl.dataflow.dflow INFO: Standard output for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/70b426ed-d278-41e9-9939-6423819e32f7_calibrate_855044_50.stdout parsl.dataflow.dflow INFO: Standard error for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/70b426ed-d278-41e9-9939-6423819e32f7_calibrate_855044_50.stderr parsl.dataflow.dflow INFO: Standard output for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/9b66bc03-4706-4333-b3f0-b409d71e2acd_characterizeImage_855044_50.stdout parsl.dataflow.dflow INFO: Standard error for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/9b66bc03-4706-4333-b3f0-b409d71e2acd_characterizeImage_855044_50.stderr parsl.dataflow.dflow INFO: Task 896 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 85cef6554c6e53b4c5ff6eb22e94b4db with result from task 896 parsl.dataflow.dflow DEBUG: Task 896 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/f0e1657f-44c0-46a5-96c9-be04538ceb0f_calibrate_855042_25.stdout parsl.dataflow.dflow INFO: Standard error for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/f0e1657f-44c0-46a5-96c9-be04538ceb0f_calibrate_855042_25.stderr parsl.dataflow.dflow INFO: Task 881 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bcdad3f205fd2ba6ad9eeeca412bba20 with result from task 881 parsl.dataflow.dflow DEBUG: Task 881 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/7f60b30c-5fcb-48d8-9300-7e7d00e94df4_calibrate_855043_46.stdout parsl.dataflow.dflow INFO: Standard error for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/7f60b30c-5fcb-48d8-9300-7e7d00e94df4_calibrate_855043_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:06 proc_lsst.multi:146 [INFO] found job 16912317 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.dataflow.dflow INFO: Task 410 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fcb51f98daa6904df8b803a8f4c1180f with result from task 410 parsl.dataflow.dflow DEBUG: Task 410 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/4b6f9edd-8931-451d-8258-c503abeb6f91_calibrate_855043_48.stdout parsl.dataflow.dflow INFO: Standard error for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/4b6f9edd-8931-451d-8258-c503abeb6f91_calibrate_855043_48.stderr parsl.dataflow.dflow INFO: Task 274 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18f851a9ceda775f158970aa00e65171 with result from task 274 parsl.dataflow.dflow DEBUG: Task 274 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 275 has memoization hash cd5e3e53535988e6c0141c3f7a5b5d26 parsl.dataflow.memoization INFO: Task 275 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 275 try 0 launched on executor multi with executor id 967 parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/31deb485-836c-4481-8900-4b3571b6eb8e_calibrate_855136_26.stdout parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/31deb485-836c-4481-8900-4b3571b6eb8e_calibrate_855136_26.stderr parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/c85999bd-bf56-470d-9af1-a1632e95d4c2_characterizeImage_855136_26.stdout parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/c85999bd-bf56-470d-9af1-a1632e95d4c2_characterizeImage_855136_26.stderr parsl.dataflow.dflow INFO: Task 884 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0aea7554b4fa411232e68dcc5ac15d65 with result from task 884 parsl.dataflow.dflow DEBUG: Task 884 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/70a3087e-ea01-4237-9720-fd2c311e4cdd_calibrate_855138_3.stdout parsl.dataflow.dflow INFO: Standard error for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/70a3087e-ea01-4237-9720-fd2c311e4cdd_calibrate_855138_3.stderr parsl.dataflow.dflow INFO: Task 154 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a5d63a71a908e2e10a1135bcf28dd83 with result from task 154 parsl.dataflow.dflow DEBUG: Task 154 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 155 has memoization hash 812f07c4ebb8a1181c3ea1148a470350 parsl.dataflow.memoization INFO: Task 155 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46814540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 155 try 0 launched on executor multi with executor id 968 parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/e45ff0a2-1822-444a-b540-12e6ed46e5a4_calibrate_855138_56.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/e45ff0a2-1822-444a-b540-12e6ed46e5a4_calibrate_855138_56.stderr parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/6ce5fb69-6137-49e5-b89d-5e80edcd4762_characterizeImage_855138_56.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/6ce5fb69-6137-49e5-b89d-5e80edcd4762_characterizeImage_855138_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:07 proc_lsst.multi:146 [INFO] found job 16912318 in provider escience_a40 proc_lsst.multi INFO: found job 16912318 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 281 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86f5b6fa80ca52666f0f46fa83fa23b1 with result from task 281 parsl.dataflow.dflow DEBUG: Task 281 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/1744c0c6-02db-4619-9dd1-3400c3bb866f_calibrate_855044_1.stdout parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/1744c0c6-02db-4619-9dd1-3400c3bb866f_calibrate_855044_1.stderr parsl.dataflow.dflow INFO: Task 253 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff1798c245d17aa52eff66ebaf62f69f with result from task 253 parsl.dataflow.dflow DEBUG: Task 253 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 254 has memoization hash 49466d15d5289b57fb35c496cbd2ca62 parsl.dataflow.memoization INFO: Task 254 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46836ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 254 try 0 launched on executor multi with executor id 969 parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/36cce5b1-6ef2-45e7-b06b-52d089636224_calibrate_855043_6.stdout parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/36cce5b1-6ef2-45e7-b06b-52d089636224_calibrate_855043_6.stderr parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/1623e654-f989-44be-9a50-4b072d4c3ff1_characterizeImage_855043_6.stdout parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/1623e654-f989-44be-9a50-4b072d4c3ff1_characterizeImage_855043_6.stderr parsl.dataflow.dflow INFO: Task 182 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f33689a8dfcb01649df9847486e6c18b with result from task 182 parsl.dataflow.dflow DEBUG: Task 182 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/4a8f38ee-ed1d-40d7-9ce2-c81ee7faae3b_calibrate_855044_58.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/4a8f38ee-ed1d-40d7-9ce2-c81ee7faae3b_calibrate_855044_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:08 proc_lsst.multi:146 [INFO] found job 16912319 in provider escience_a40 proc_lsst.multi INFO: found job 16912319 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 164 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b2f1d90a5866200c59f5eea0d5f73b37 with result from task 164 parsl.dataflow.dflow DEBUG: Task 164 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/6401e3ce-0afd-4b57-8a70-b7e10a6e50d5_calibrate_855042_35.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/6401e3ce-0afd-4b57-8a70-b7e10a6e50d5_calibrate_855042_35.stderr parsl.dataflow.dflow INFO: Task 1048 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b98af4b47132c029fc5f14452a56943 with result from task 1048 parsl.dataflow.dflow DEBUG: Task 1048 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1049 has memoization hash f24e9ffe243adede1f632750e76b496b parsl.dataflow.memoization INFO: Task 1049 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45543420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1049 try 0 launched on executor multi with executor id 970 parsl.dataflow.dflow INFO: Standard output for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/21c4298a-766b-418a-896f-33e54fe89c84_calibrate_855137_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/21c4298a-766b-418a-896f-33e54fe89c84_calibrate_855137_9.stderr parsl.dataflow.dflow INFO: Standard output for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/bf774540-4587-4429-871e-307b3e84f4b3_characterizeImage_855137_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/bf774540-4587-4429-871e-307b3e84f4b3_characterizeImage_855137_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:09 proc_lsst.multi:146 [INFO] found job 16912320 in provider escience_a40 proc_lsst.multi INFO: found job 16912320 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 470 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ab7a82ebf9d8499547c432fce98a8b6 with result from task 470 parsl.dataflow.dflow DEBUG: Task 470 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/76cd631a-4524-488f-9532-57081b1f0cfe_calibrate_855044_15.stdout parsl.dataflow.dflow INFO: Standard error for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/76cd631a-4524-488f-9532-57081b1f0cfe_calibrate_855044_15.stderr parsl.dataflow.dflow INFO: Task 14 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 520b344952e0a71d1b83f8a5354e51d9 with result from task 14 parsl.dataflow.dflow DEBUG: Task 14 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/e2d20e37-b84b-4151-ba11-3e35372bb4bd_calibrate_855044_4.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/e2d20e37-b84b-4151-ba11-3e35372bb4bd_calibrate_855044_4.stderr parsl.dataflow.dflow INFO: Task 821 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 406841e6406f277b7594f7d855f0905f with result from task 821 parsl.dataflow.dflow DEBUG: Task 821 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/2fe5a49e-48ac-452a-81b1-b4c933a85941_calibrate_855042_58.stdout parsl.dataflow.dflow INFO: Standard error for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/2fe5a49e-48ac-452a-81b1-b4c933a85941_calibrate_855042_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:10 proc_lsst.multi:146 [INFO] found job 16912321 in provider escience_a40 proc_lsst.multi INFO: found job 16912321 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 1069 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ddef1343a806f2ce22a60da125fc193 with result from task 1069 parsl.dataflow.dflow DEBUG: Task 1069 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1070 has memoization hash 072a03a5183bf0d28047f87d5482e2ce parsl.dataflow.memoization INFO: Task 1070 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45392f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1070 try 0 launched on executor multi with executor id 971 parsl.dataflow.dflow INFO: Standard output for task 1070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/e3e535c0-44e8-4ded-89dc-3351b7c721a1_calibrate_855138_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/e3e535c0-44e8-4ded-89dc-3351b7c721a1_calibrate_855138_36.stderr parsl.dataflow.dflow INFO: Standard output for task 1069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/8fbb9451-4989-4975-8402-360d1c4d67df_characterizeImage_855138_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/8fbb9451-4989-4975-8402-360d1c4d67df_characterizeImage_855138_36.stderr parsl.dataflow.dflow INFO: Task 737 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e46a3e92ce7c98cd77604cd77a39366 with result from task 737 parsl.dataflow.dflow DEBUG: Task 737 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/3f641e5d-4f9b-48b3-8a92-14b66091cea6_calibrate_855044_6.stdout parsl.dataflow.dflow INFO: Standard error for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/3f641e5d-4f9b-48b3-8a92-14b66091cea6_calibrate_855044_6.stderr parsl.dataflow.dflow INFO: Task 169 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9158f6f3a28498edde514bd5f4316e5b with result from task 169 parsl.dataflow.dflow DEBUG: Task 169 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 170 has memoization hash 3bc1aaaf48a4f010aeeca3ce86871c38 parsl.dataflow.memoization INFO: Task 170 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46815300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 170 try 0 launched on executor multi with executor id 972 parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/5a082a25-43a5-4799-90cf-d4d80f9e5c6e_calibrate_855042_39.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/5a082a25-43a5-4799-90cf-d4d80f9e5c6e_calibrate_855042_39.stderr parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/f478c8e6-13d6-4368-8938-660b253e16d3_characterizeImage_855042_39.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/f478c8e6-13d6-4368-8938-660b253e16d3_characterizeImage_855042_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:10 proc_lsst.multi:146 [INFO] found job 16912322 in provider escience_a40 proc_lsst.multi INFO: found job 16912322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 190 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c30e9ba708ae6eb2c52972eaff08763 with result from task 190 parsl.dataflow.dflow DEBUG: Task 190 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 191 has memoization hash 34fc7ef56ed9875808b286d2c363d106 parsl.dataflow.memoization INFO: Task 191 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa468168e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 191 try 0 launched on executor multi with executor id 973 parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/91f7512b-941c-4cba-81ed-498e70d0ff7e_calibrate_855044_60.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/91f7512b-941c-4cba-81ed-498e70d0ff7e_calibrate_855044_60.stderr parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/2a2c6661-715b-4b61-9883-5e6a9da92bd3_characterizeImage_855044_60.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/2a2c6661-715b-4b61-9883-5e6a9da92bd3_characterizeImage_855044_60.stderr parsl.dataflow.dflow INFO: Task 158 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 76f38a5b15e95d95f2493a52c3b51854 with result from task 158 parsl.dataflow.dflow DEBUG: Task 158 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/ca87c467-32bc-45db-9a62-6fdc570d71ff_calibrate_855043_55.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/ca87c467-32bc-45db-9a62-6fdc570d71ff_calibrate_855043_55.stderr parsl.dataflow.dflow INFO: Task 338 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf6cfa249a1ef4867e9110913ebbb141 with result from task 338 parsl.dataflow.dflow DEBUG: Task 338 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/f5132bfe-afe5-4892-9de2-5098a0ab0a16_calibrate_855043_5.stdout parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/f5132bfe-afe5-4892-9de2-5098a0ab0a16_calibrate_855043_5.stderr parsl.dataflow.dflow INFO: Task 298 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 481479a7d79c0c16eb43ca2e298ee348 with result from task 298 parsl.dataflow.dflow DEBUG: Task 298 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 299 has memoization hash e2ade6c1732f8967c5b7bef827d0d355 parsl.dataflow.memoization INFO: Task 299 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 299 try 0 launched on executor multi with executor id 974 parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/c99ecf73-e117-476e-941d-cd954cd7a905_calibrate_855137_43.stdout parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/c99ecf73-e117-476e-941d-cd954cd7a905_calibrate_855137_43.stderr parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/2196edd3-8030-41f7-a046-36643ff87b2a_characterizeImage_855137_43.stdout parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/2196edd3-8030-41f7-a046-36643ff87b2a_characterizeImage_855137_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:11 proc_lsst.multi:146 [INFO] found job 16912323 in provider escience_a40 proc_lsst.multi INFO: found job 16912323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 541 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dbe37818638973db44c0eb5cc6c0b07d with result from task 541 parsl.dataflow.dflow DEBUG: Task 541 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 542 has memoization hash b9a73d5b0cf723e8fc6cf5507b18c942 parsl.dataflow.memoization INFO: Task 542 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45701080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 542 try 0 launched on executor multi with executor id 975 parsl.dataflow.dflow INFO: Standard output for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/26899f1f-ba0f-4565-ba4d-44eaa1e1a54d_calibrate_855044_20.stdout parsl.dataflow.dflow INFO: Standard error for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/26899f1f-ba0f-4565-ba4d-44eaa1e1a54d_calibrate_855044_20.stderr parsl.dataflow.dflow INFO: Standard output for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/2b8da6f1-b46e-4f6e-a20e-1fbfdfbbf610_characterizeImage_855044_20.stdout parsl.dataflow.dflow INFO: Standard error for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/2b8da6f1-b46e-4f6e-a20e-1fbfdfbbf610_characterizeImage_855044_20.stderr parsl.dataflow.dflow INFO: Task 499 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce2bbb321533d9fcde9b0862d820e545 with result from task 499 parsl.dataflow.dflow DEBUG: Task 499 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 500 has memoization hash 92fd4a8545d2443465f79dd730fdaef4 parsl.dataflow.memoization INFO: Task 500 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456ce020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 500 try 0 launched on executor multi with executor id 976 parsl.dataflow.dflow INFO: Standard output for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/40a86c7d-dd94-4b9b-b59b-e5ee20411a8d_calibrate_855137_28.stdout parsl.dataflow.dflow INFO: Standard error for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/40a86c7d-dd94-4b9b-b59b-e5ee20411a8d_calibrate_855137_28.stderr parsl.dataflow.dflow INFO: Standard output for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/d08733f0-f3c6-4236-bd93-8d2e204c4db1_characterizeImage_855137_28.stdout parsl.dataflow.dflow INFO: Standard error for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/d08733f0-f3c6-4236-bd93-8d2e204c4db1_characterizeImage_855137_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:12 proc_lsst.multi:146 [INFO] found job 16912324 in provider escience_a40 proc_lsst.multi INFO: found job 16912324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 955 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5cee067dece8740030789b3c9469aad8 with result from task 955 parsl.dataflow.dflow DEBUG: Task 955 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 956 has memoization hash fcf1668f9bb4ffa3d724a31eb60853c1 parsl.dataflow.memoization INFO: Task 956 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454fa520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 956 try 0 launched on executor multi with executor id 977 parsl.dataflow.dflow INFO: Standard output for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/19250f4e-6071-4aef-9f1d-c04346d79eaa_calibrate_855138_7.stdout parsl.dataflow.dflow INFO: Standard error for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/19250f4e-6071-4aef-9f1d-c04346d79eaa_calibrate_855138_7.stderr parsl.dataflow.dflow INFO: Standard output for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/d6440207-a7f8-4b3d-a85f-e61cb04f150d_characterizeImage_855138_7.stdout parsl.dataflow.dflow INFO: Standard error for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/d6440207-a7f8-4b3d-a85f-e61cb04f150d_characterizeImage_855138_7.stderr parsl.dataflow.dflow INFO: Task 340 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de8a9f0b2009bb63b77074655e08227d with result from task 340 parsl.dataflow.dflow DEBUG: Task 340 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 341 has memoization hash cd763083c68b271396d0ba3da5e8a641 parsl.dataflow.memoization INFO: Task 341 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa468785e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 341 try 0 launched on executor multi with executor id 978 parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/eeb90391-ad6e-4121-b8ae-9efab92cea3b_calibrate_855137_47.stdout parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/eeb90391-ad6e-4121-b8ae-9efab92cea3b_calibrate_855137_47.stderr parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/fb187de7-a51f-43af-89ee-36656426f29a_characterizeImage_855137_47.stdout parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/fb187de7-a51f-43af-89ee-36656426f29a_characterizeImage_855137_47.stderr parsl.dataflow.dflow INFO: Task 794 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6cbe34e1c5557c879b1c17b6edef7b0 with result from task 794 parsl.dataflow.dflow DEBUG: Task 794 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/efdd8dde-237a-4786-ba7d-e487ef81af4c_calibrate_855043_49.stdout parsl.dataflow.dflow INFO: Standard error for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/efdd8dde-237a-4786-ba7d-e487ef81af4c_calibrate_855043_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:13 proc_lsst.multi:146 [INFO] found job 16912325 in provider escience_a40 proc_lsst.multi INFO: found job 16912325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 146 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eab2957b36e32d07261a5bf4c2d68728 with result from task 146 parsl.dataflow.dflow DEBUG: Task 146 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/b98323b9-6f9b-4c14-bfaf-6e6e24fbb528_calibrate_855042_45.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/b98323b9-6f9b-4c14-bfaf-6e6e24fbb528_calibrate_855042_45.stderr parsl.dataflow.dflow INFO: Task 1066 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 30ee99cea6ac39d58b868193ed6a56fc with result from task 1066 parsl.dataflow.dflow DEBUG: Task 1066 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1067 has memoization hash ef0676ff8fa0f13ed20bf5368a82c168 parsl.dataflow.memoization INFO: Task 1067 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45393420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1067 try 0 launched on executor multi with executor id 979 parsl.dataflow.dflow INFO: Standard output for task 1067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/ab1b1548-4c9d-41d6-b7bf-bae8cc61117e_calibrate_855138_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/ab1b1548-4c9d-41d6-b7bf-bae8cc61117e_calibrate_855138_19.stderr parsl.dataflow.dflow INFO: Standard output for task 1066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/104e703f-17b1-472c-8d4f-2610345dee80_characterizeImage_855138_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/104e703f-17b1-472c-8d4f-2610345dee80_characterizeImage_855138_19.stderr parsl.dataflow.dflow INFO: Task 511 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c403083d8a8dbd4b21e8b4ef2bcc0bac with result from task 511 parsl.dataflow.dflow DEBUG: Task 511 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 512 has memoization hash 043cef6d221aed96de7b3d78f012b354 parsl.dataflow.memoization INFO: Task 512 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456cfc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 512 try 0 launched on executor multi with executor id 980 parsl.dataflow.dflow INFO: Standard output for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/977fe3a5-819f-4f77-a8c0-de5809e5055a_calibrate_855042_44.stdout parsl.dataflow.dflow INFO: Standard error for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/977fe3a5-819f-4f77-a8c0-de5809e5055a_calibrate_855042_44.stderr parsl.dataflow.dflow INFO: Standard output for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/11432293-3e64-4f5a-ae6a-0051f04af7ef_characterizeImage_855042_44.stdout parsl.dataflow.dflow INFO: Standard error for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/11432293-3e64-4f5a-ae6a-0051f04af7ef_characterizeImage_855042_44.stderr parsl.dataflow.dflow INFO: Task 505 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2858b74e355a64f4a47de945ff44741 with result from task 505 parsl.dataflow.dflow DEBUG: Task 505 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 506 has memoization hash 969e76bf2ac609617ab6c64472869dff parsl.dataflow.memoization INFO: Task 506 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456ce5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 506 try 0 launched on executor multi with executor id 981 parsl.dataflow.dflow INFO: Standard output for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/163b3d5c-e9d6-44cf-886c-d10f11c2253a_calibrate_855042_15.stdout parsl.dataflow.dflow INFO: Standard error for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/163b3d5c-e9d6-44cf-886c-d10f11c2253a_calibrate_855042_15.stderr parsl.dataflow.dflow INFO: Standard output for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/4a37ec0c-d957-4b53-a0ff-fbdc15050e92_characterizeImage_855042_15.stdout parsl.dataflow.dflow INFO: Standard error for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/4a37ec0c-d957-4b53-a0ff-fbdc15050e92_characterizeImage_855042_15.stderr parsl.dataflow.dflow INFO: Task 568 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33cc4bed79fa4a1ac43e1ded7bbb13a4 with result from task 568 parsl.dataflow.dflow DEBUG: Task 568 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 569 has memoization hash a26e877d4fc22006e6efe3b103d4f2c5 parsl.dataflow.memoization INFO: Task 569 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa457034c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 569 try 0 launched on executor multi with executor id 982 parsl.dataflow.dflow INFO: Standard output for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/976ba8fe-13f2-4bc0-af97-a129a85e64c7_calibrate_855043_50.stdout parsl.dataflow.dflow INFO: Standard error for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/976ba8fe-13f2-4bc0-af97-a129a85e64c7_calibrate_855043_50.stderr parsl.dataflow.dflow INFO: Standard output for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/96036120-8b78-4714-9391-8510d32c9df3_characterizeImage_855043_50.stdout parsl.dataflow.dflow INFO: Standard error for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/96036120-8b78-4714-9391-8510d32c9df3_characterizeImage_855043_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:13 proc_lsst.multi:146 [INFO] found job 16912326 in provider escience_a40 proc_lsst.multi INFO: found job 16912326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 79 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e752a7b4cb8c92afeb74287cc1fdc28 with result from task 79 parsl.dataflow.dflow DEBUG: Task 79 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 80 has memoization hash a5b00c3ff7f27688d2f6f6a659a332a1 parsl.dataflow.memoization INFO: Task 80 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d34c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 80 try 0 launched on executor multi with executor id 983 parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/0fdc6710-c726-464f-bcd3-bbd2bd4df789_calibrate_855136_60.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/0fdc6710-c726-464f-bcd3-bbd2bd4df789_calibrate_855136_60.stderr parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/7c458fee-c571-466e-b4c7-e9f47152600e_characterizeImage_855136_60.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/7c458fee-c571-466e-b4c7-e9f47152600e_characterizeImage_855136_60.stderr parsl.dataflow.dflow INFO: Task 856 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 75bc24e853dba4b852ad103fd3434a8f with result from task 856 parsl.dataflow.dflow DEBUG: Task 856 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 857 has memoization hash 97f1fcfa46a8903f19373dfb7254bffb parsl.dataflow.memoization INFO: Task 857 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45499580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 857 try 0 launched on executor multi with executor id 984 parsl.dataflow.dflow INFO: Standard output for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/8acf325b-4e50-4a67-80a8-fa72b454d897_calibrate_855043_43.stdout parsl.dataflow.dflow INFO: Standard error for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/8acf325b-4e50-4a67-80a8-fa72b454d897_calibrate_855043_43.stderr parsl.dataflow.dflow INFO: Standard output for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/d8ce44fb-0353-4fd5-84a6-7e824d54df22_characterizeImage_855043_43.stdout parsl.dataflow.dflow INFO: Standard error for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/d8ce44fb-0353-4fd5-84a6-7e824d54df22_characterizeImage_855043_43.stderr parsl.dataflow.dflow INFO: Task 250 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e997321db0d6452cae1e8f6ac6655d02 with result from task 250 parsl.dataflow.dflow DEBUG: Task 250 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 251 has memoization hash a0c96ed2eac400951fe54832109dba69 parsl.dataflow.memoization INFO: Task 251 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa468367a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 251 try 0 launched on executor multi with executor id 985 parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/65984a32-9531-4f10-ad22-71132728401b_calibrate_855136_36.stdout parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/65984a32-9531-4f10-ad22-71132728401b_calibrate_855136_36.stderr parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/48a4cff3-1b16-498c-85a0-d1af0b330f73_characterizeImage_855136_36.stdout parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/48a4cff3-1b16-498c-85a0-d1af0b330f73_characterizeImage_855136_36.stderr parsl.dataflow.dflow INFO: Task 56 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de9049970472bbb4ec1d8a10032f40ed with result from task 56 parsl.dataflow.dflow DEBUG: Task 56 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/d2c6f2f9-f3d0-4295-9171-05793d42a4d7_calibrate_855044_8.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/d2c6f2f9-f3d0-4295-9171-05793d42a4d7_calibrate_855044_8.stderr parsl.dataflow.dflow INFO: Task 101 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 352a6bcabf2c88b2c2a358167558afee with result from task 101 parsl.dataflow.dflow DEBUG: Task 101 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/0d931dc0-3286-4301-aef0-20e858f502ee_calibrate_855042_49.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/0d931dc0-3286-4301-aef0-20e858f502ee_calibrate_855042_49.stderr parsl.dataflow.dflow INFO: Task 293 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 44df8a7460caf03a6b967ae33bb9f18b with result from task 293 parsl.dataflow.dflow DEBUG: Task 293 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/fb852705-b881-4a34-8873-308beefcd0e0_calibrate_855044_12.stdout parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/fb852705-b881-4a34-8873-308beefcd0e0_calibrate_855044_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:14 proc_lsst.multi:146 [INFO] found job 16912327 in provider escience_a40 proc_lsst.multi INFO: found job 16912327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 431 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65d62ecc7e404add615ea379e142920a with result from task 431 parsl.dataflow.dflow DEBUG: Task 431 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/701f1c83-c2f9-45ec-95d7-98477fca9cf8_calibrate_855043_4.stdout parsl.dataflow.dflow INFO: Standard error for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/701f1c83-c2f9-45ec-95d7-98477fca9cf8_calibrate_855043_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:15 proc_lsst.multi:146 [INFO] found job 16912328 in provider escience_a40 proc_lsst.multi INFO: found job 16912328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 659 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80725b2969b3eee1006e5a9af3b08a10 with result from task 659 parsl.dataflow.dflow DEBUG: Task 659 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/26f6e728-5333-405c-9ce5-dc71236d69c0_calibrate_855042_36.stdout parsl.dataflow.dflow INFO: Standard error for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/26f6e728-5333-405c-9ce5-dc71236d69c0_calibrate_855042_36.stderr parsl.dataflow.dflow INFO: Task 8 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 431c8786028091c30097524d1eaaa862 with result from task 8 parsl.dataflow.dflow DEBUG: Task 8 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/07a68777-e012-4485-a851-a72643564d0b_calibrate_855044_41.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/07a68777-e012-4485-a851-a72643564d0b_calibrate_855044_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:16 proc_lsst.multi:146 [INFO] found job 16912329 in provider escience_a40 proc_lsst.multi INFO: found job 16912329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 566 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de77b89c5ded6d97c9d15eb2b0eed6b0 with result from task 566 parsl.dataflow.dflow DEBUG: Task 566 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/8d5b2d58-6bce-43b5-b8e3-a97556aef937_calibrate_855136_7.stdout parsl.dataflow.dflow INFO: Standard error for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/8d5b2d58-6bce-43b5-b8e3-a97556aef937_calibrate_855136_7.stderr parsl.dataflow.dflow INFO: Task 92 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 571c563f9d0dc517510e8eceb61ee335 with result from task 92 parsl.dataflow.dflow DEBUG: Task 92 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/39e58f0f-a79d-46bb-a573-c1a615bc8663_calibrate_855043_19.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/39e58f0f-a79d-46bb-a573-c1a615bc8663_calibrate_855043_19.stderr parsl.dataflow.dflow INFO: Task 110 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 19dd6640b6da2ed60241ebd12f129bdb with result from task 110 parsl.dataflow.dflow DEBUG: Task 110 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/bf8a19f7-74d8-44e4-b505-a3c3a379f8a7_calibrate_855044_21.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/bf8a19f7-74d8-44e4-b505-a3c3a379f8a7_calibrate_855044_21.stderr parsl.dataflow.dflow INFO: Task 428 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49a703e1f42f140b616240290c54eadf with result from task 428 parsl.dataflow.dflow DEBUG: Task 428 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/bd6ab114-a354-4d11-82ed-a64d9fcc4bae_calibrate_855136_25.stdout parsl.dataflow.dflow INFO: Standard error for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/bd6ab114-a354-4d11-82ed-a64d9fcc4bae_calibrate_855136_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:17 proc_lsst.multi:146 [INFO] found job 16912330 in provider escience_a40 proc_lsst.multi INFO: found job 16912330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 598 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f728d28cbdc9b5b37ff2a55a19e980ff with result from task 598 parsl.dataflow.dflow DEBUG: Task 598 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 599 has memoization hash cf6558b039db453d16036087a5b4e241 parsl.dataflow.memoization INFO: Task 599 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa457409a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 599 try 0 launched on executor multi with executor id 986 parsl.dataflow.dflow INFO: Standard output for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/f71abd10-fca0-40ea-87f5-26a475dd79d6_calibrate_855042_33.stdout parsl.dataflow.dflow INFO: Standard error for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/f71abd10-fca0-40ea-87f5-26a475dd79d6_calibrate_855042_33.stderr parsl.dataflow.dflow INFO: Standard output for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/6cacb009-b5f7-47f6-92fd-064222895fec_characterizeImage_855042_33.stdout parsl.dataflow.dflow INFO: Standard error for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/6cacb009-b5f7-47f6-92fd-064222895fec_characterizeImage_855042_33.stderr parsl.dataflow.dflow INFO: Task 733 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2b18739d0bc8a780e844a0231176129 with result from task 733 parsl.dataflow.dflow DEBUG: Task 733 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 734 has memoization hash e08de78c367d8f798e95b81f0b9353ba parsl.dataflow.memoization INFO: Task 734 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d22a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 734 try 0 launched on executor multi with executor id 987 parsl.dataflow.dflow INFO: Standard output for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/8030e4c1-4790-422c-a7cd-177aab6b8d44_calibrate_855136_29.stdout parsl.dataflow.dflow INFO: Standard error for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/8030e4c1-4790-422c-a7cd-177aab6b8d44_calibrate_855136_29.stderr parsl.dataflow.dflow INFO: Standard output for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/912de327-6e6b-45e9-bb53-7af41f1e4b5b_characterizeImage_855136_29.stdout parsl.dataflow.dflow INFO: Standard error for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/912de327-6e6b-45e9-bb53-7af41f1e4b5b_characterizeImage_855136_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:18 proc_lsst.multi:146 [INFO] found job 16912331 in provider escience_a40 proc_lsst.multi INFO: found job 16912331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 443 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e8edfae81bc13a2e36c5e9786c65684 with result from task 443 parsl.dataflow.dflow DEBUG: Task 443 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/52e64ecc-70d6-47b4-a2ea-4417e5340936_calibrate_855138_1.stdout parsl.dataflow.dflow INFO: Standard error for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/52e64ecc-70d6-47b4-a2ea-4417e5340936_calibrate_855138_1.stderr parsl.dataflow.dflow INFO: Task 614 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3624f130a3d64463ec9f428def640b9 with result from task 614 parsl.dataflow.dflow DEBUG: Task 614 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/d25359d1-eff0-4228-b21c-e8b78e55179d_calibrate_855044_29.stdout parsl.dataflow.dflow INFO: Standard error for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/d25359d1-eff0-4228-b21c-e8b78e55179d_calibrate_855044_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:18 proc_lsst.multi:146 [INFO] found job 16912332 in provider escience_a40 proc_lsst.multi INFO: found job 16912332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 29 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa3920af2d43c023ca412b830fd4127b with result from task 29 parsl.dataflow.dflow DEBUG: Task 29 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/b9cffdc3-2530-49eb-97ec-41dea25fb017_calibrate_855137_19.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/b9cffdc3-2530-49eb-97ec-41dea25fb017_calibrate_855137_19.stderr parsl.dataflow.dflow INFO: Task 946 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d739f5c00e671c8ba55842105e1fffd1 with result from task 946 parsl.dataflow.dflow DEBUG: Task 946 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 947 has memoization hash d25ab06265ed52398ff20e67f2a01865 parsl.dataflow.memoization INFO: Task 947 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454faca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 947 try 0 launched on executor multi with executor id 988 parsl.dataflow.dflow INFO: Standard output for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/d9821108-412f-4b6d-b861-ec692152f986_calibrate_855136_22.stdout parsl.dataflow.dflow INFO: Standard error for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/d9821108-412f-4b6d-b861-ec692152f986_calibrate_855136_22.stderr parsl.dataflow.dflow INFO: Standard output for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/f18a6e67-38ad-4947-8533-fb1868ed225c_characterizeImage_855136_22.stdout parsl.dataflow.dflow INFO: Standard error for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/f18a6e67-38ad-4947-8533-fb1868ed225c_characterizeImage_855136_22.stderr parsl.dataflow.dflow INFO: Task 83 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 494920ece182dfc0b7a1e6e082e495a7 with result from task 83 parsl.dataflow.dflow DEBUG: Task 83 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/d2ee6f41-45d3-4b59-ace2-b0d84370fe4d_calibrate_855138_8.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/d2ee6f41-45d3-4b59-ace2-b0d84370fe4d_calibrate_855138_8.stderr parsl.dataflow.dflow INFO: Task 608 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55033658773cefd60667d29cd7c7f7e5 with result from task 608 parsl.dataflow.dflow DEBUG: Task 608 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/6667b1f9-161f-49b4-aed2-dc73d7abe15d_calibrate_855138_5.stdout parsl.dataflow.dflow INFO: Standard error for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/6667b1f9-161f-49b4-aed2-dc73d7abe15d_calibrate_855138_5.stderr parsl.dataflow.dflow INFO: Task 346 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de5d53d135a5a71bec336ffb260c2831 with result from task 346 parsl.dataflow.dflow DEBUG: Task 346 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 347 has memoization hash d5a4b86d30e4e07f32e9c524adbb792f parsl.dataflow.memoization INFO: Task 347 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46878c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 347 try 0 launched on executor multi with executor id 989 parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/ff1d6b49-fd3f-4969-80a4-d085692f6d38_calibrate_855043_25.stdout parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/ff1d6b49-fd3f-4969-80a4-d085692f6d38_calibrate_855043_25.stderr parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/aa5d6fda-321c-4e44-b9f2-76d10a9df141_characterizeImage_855043_25.stdout parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/aa5d6fda-321c-4e44-b9f2-76d10a9df141_characterizeImage_855043_25.stderr parsl.dataflow.dflow INFO: Task 434 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f0ab61dd64729c41f1991d381d3a8f8c with result from task 434 parsl.dataflow.dflow DEBUG: Task 434 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/1b2819fd-ea0d-4b23-bd34-2dca58de5e0a_calibrate_855042_27.stdout parsl.dataflow.dflow INFO: Standard error for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/1b2819fd-ea0d-4b23-bd34-2dca58de5e0a_calibrate_855042_27.stderr parsl.dataflow.dflow INFO: Task 389 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5b8d45db34ae0d81c36eff3cb20bcf8 with result from task 389 parsl.dataflow.dflow DEBUG: Task 389 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/d28fdede-3ca2-4fb8-ae66-9c64d5f82276_calibrate_855042_51.stdout parsl.dataflow.dflow INFO: Standard error for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/d28fdede-3ca2-4fb8-ae66-9c64d5f82276_calibrate_855042_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:19 proc_lsst.multi:146 [INFO] found job 16912333 in provider escience_a40 proc_lsst.multi INFO: found job 16912333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 869 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7fbc56499d88e1f3be308c2ef9607306 with result from task 869 parsl.dataflow.dflow DEBUG: Task 869 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/ebd9b366-51d3-4259-a2a7-68d709e50357_calibrate_855044_13.stdout parsl.dataflow.dflow INFO: Standard error for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/ebd9b366-51d3-4259-a2a7-68d709e50357_calibrate_855044_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:20 proc_lsst.multi:146 [INFO] found job 16912334 in provider escience_a40 proc_lsst.multi INFO: found job 16912334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 134 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09228c2ba7519c78995eb088b447c8b7 with result from task 134 parsl.dataflow.dflow DEBUG: Task 134 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/759b5834-92e5-4c07-92cd-e0162e423dff_calibrate_855044_45.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/759b5834-92e5-4c07-92cd-e0162e423dff_calibrate_855044_45.stderr parsl.dataflow.dflow INFO: Task 371 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 11a2c733b159598ab1fad404a7505aa6 with result from task 371 parsl.dataflow.dflow DEBUG: Task 371 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/94f54cfe-66c6-414c-9416-e6156e3214a2_calibrate_855137_3.stdout parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/94f54cfe-66c6-414c-9416-e6156e3214a2_calibrate_855137_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:21 proc_lsst.multi:146 [INFO] found job 16912335 in provider escience_a40 proc_lsst.multi INFO: found job 16912335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 850 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2a1dc0301cd755be9f779033a62f6f5 with result from task 850 parsl.dataflow.dflow DEBUG: Task 850 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 851 has memoization hash 6e9dc6975329cf2ea4c98cb4a16be16d parsl.dataflow.memoization INFO: Task 851 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45665d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 851 try 0 launched on executor multi with executor id 990 parsl.dataflow.dflow INFO: Standard output for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/3a849d7d-aa12-4d69-9c13-e8b718809809_calibrate_855042_54.stdout parsl.dataflow.dflow INFO: Standard error for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/3a849d7d-aa12-4d69-9c13-e8b718809809_calibrate_855042_54.stderr parsl.dataflow.dflow INFO: Standard output for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/21d62c68-aa76-428e-bbe8-3ea72f81978f_characterizeImage_855042_54.stdout parsl.dataflow.dflow INFO: Standard error for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/21d62c68-aa76-428e-bbe8-3ea72f81978f_characterizeImage_855042_54.stderr parsl.dataflow.dflow INFO: Task 707 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae31529b58d725e0a566abc6bf6ecf7e with result from task 707 parsl.dataflow.dflow DEBUG: Task 707 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/f06d5736-5ca9-4c5d-878d-baea609a23dc_calibrate_855138_41.stdout parsl.dataflow.dflow INFO: Standard error for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/f06d5736-5ca9-4c5d-878d-baea609a23dc_calibrate_855138_41.stderr parsl.dataflow.dflow INFO: Task 268 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c73f116277d182ffe20ddc16198ff5d3 with result from task 268 parsl.dataflow.dflow DEBUG: Task 268 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 269 has memoization hash d5027c00cde79f77f334374b79512a80 parsl.dataflow.memoization INFO: Task 269 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46837a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 269 try 0 launched on executor multi with executor id 991 parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/a2fdbdea-4929-4673-aaf1-53fe4d7b424e_calibrate_855136_14.stdout parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/a2fdbdea-4929-4673-aaf1-53fe4d7b424e_calibrate_855136_14.stderr parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/fdf7df9e-8bbc-4997-8fda-09188ab365a4_characterizeImage_855136_14.stdout parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/fdf7df9e-8bbc-4997-8fda-09188ab365a4_characterizeImage_855136_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:22 proc_lsst.multi:146 [INFO] found job 16912336 in provider escience_a40 proc_lsst.multi INFO: found job 16912336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 493 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55091645fbc2af2e51fec5b2b8ee6aaf with result from task 493 parsl.dataflow.dflow DEBUG: Task 493 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 494 has memoization hash a0b5adfc0a20db257afa3d2d457c6fa2 parsl.dataflow.memoization INFO: Task 494 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456cd9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 494 try 0 launched on executor multi with executor id 992 parsl.dataflow.dflow INFO: Standard output for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/f5880d76-1cbb-4b7e-a3df-856a4464ca1c_calibrate_855043_3.stdout parsl.dataflow.dflow INFO: Standard error for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/f5880d76-1cbb-4b7e-a3df-856a4464ca1c_calibrate_855043_3.stderr parsl.dataflow.dflow INFO: Standard output for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/7023f654-6de7-447a-ba18-ec2e285d8ffc_characterizeImage_855043_3.stdout parsl.dataflow.dflow INFO: Standard error for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/7023f654-6de7-447a-ba18-ec2e285d8ffc_characterizeImage_855043_3.stderr parsl.dataflow.dflow INFO: Task 113 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb9a9e3613ee0fe04d43e8aa53dcddca with result from task 113 parsl.dataflow.dflow DEBUG: Task 113 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/95bfd1b2-2a95-43f3-ba18-e25b6e3d544d_calibrate_855138_21.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/95bfd1b2-2a95-43f3-ba18-e25b6e3d544d_calibrate_855138_21.stderr parsl.dataflow.dflow INFO: Task 763 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eba77e20e37a32b66a51fa94fe977dea with result from task 763 parsl.dataflow.dflow DEBUG: Task 763 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 764 has memoization hash fc3b9f5ceb82c0eca73c8a75c13dec89 parsl.dataflow.memoization INFO: Task 764 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d3420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 764 try 0 launched on executor multi with executor id 993 parsl.dataflow.dflow INFO: Standard output for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/d89ca6d1-9a94-4e02-8cc9-d0f0d1f314fb_calibrate_855137_18.stdout parsl.dataflow.dflow INFO: Standard error for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/d89ca6d1-9a94-4e02-8cc9-d0f0d1f314fb_calibrate_855137_18.stderr parsl.dataflow.dflow INFO: Standard output for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/ed3f38e4-304a-46f6-b4b8-93713ae9c2e7_characterizeImage_855137_18.stdout parsl.dataflow.dflow INFO: Standard error for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/ed3f38e4-304a-46f6-b4b8-93713ae9c2e7_characterizeImage_855137_18.stderr parsl.dataflow.dflow INFO: Task 514 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5f0f32698c228dbc6e3e6378049587e with result from task 514 parsl.dataflow.dflow DEBUG: Task 514 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 515 has memoization hash e8c83da50526ebc0bd1ebad29bb55160 parsl.dataflow.memoization INFO: Task 515 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456ce700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 515 try 0 launched on executor multi with executor id 994 parsl.dataflow.dflow INFO: Standard output for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/6384b77d-ecf1-45c1-8e88-323bbb918e9c_calibrate_855043_41.stdout parsl.dataflow.dflow INFO: Standard error for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/6384b77d-ecf1-45c1-8e88-323bbb918e9c_calibrate_855043_41.stderr parsl.dataflow.dflow INFO: Standard output for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/2e173f01-60d3-4f5d-9565-f0dd0bd841d2_characterizeImage_855043_41.stdout parsl.dataflow.dflow INFO: Standard error for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/2e173f01-60d3-4f5d-9565-f0dd0bd841d2_characterizeImage_855043_41.stderr parsl.dataflow.dflow INFO: Task 740 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 551645b7709bf9d161ab85225f326e4a with result from task 740 parsl.dataflow.dflow DEBUG: Task 740 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/c7477635-d3ba-4b51-ba27-2e538e100f17_calibrate_855136_50.stdout parsl.dataflow.dflow INFO: Standard error for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/c7477635-d3ba-4b51-ba27-2e538e100f17_calibrate_855136_50.stderr parsl.dataflow.dflow INFO: Task 116 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a61e7d9e082307a37210e0aebfff03b with result from task 116 parsl.dataflow.dflow DEBUG: Task 116 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/c333f88c-43bb-4453-91e1-4a3e6a222aa4_calibrate_855136_37.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/c333f88c-43bb-4453-91e1-4a3e6a222aa4_calibrate_855136_37.stderr parsl.dataflow.dflow INFO: Task 304 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1d72151abb9626448c8320a1353a6d0 with result from task 304 parsl.dataflow.dflow DEBUG: Task 304 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 305 has memoization hash 3475df9502128e88168edb80d5bddd64 parsl.dataflow.memoization INFO: Task 305 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 305 try 0 launched on executor multi with executor id 995 parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/fda72563-cbab-49a9-9437-9fb49253d6ac_calibrate_855044_39.stdout parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/fda72563-cbab-49a9-9437-9fb49253d6ac_calibrate_855044_39.stderr parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/178c32e4-7522-418d-8d08-c6be0d08a4c9_characterizeImage_855044_39.stdout parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/178c32e4-7522-418d-8d08-c6be0d08a4c9_characterizeImage_855044_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:23 proc_lsst.multi:146 [INFO] found job 16912337 in provider escience_a40 proc_lsst.multi INFO: found job 16912337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 559 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0bc2ecbec6bfee3f800e01c51610afe3 with result from task 559 parsl.dataflow.dflow DEBUG: Task 559 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 560 has memoization hash 10e7485419ed6a7e7d019bcbde1e7b6e parsl.dataflow.memoization INFO: Task 560 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45702480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 560 try 0 launched on executor multi with executor id 996 parsl.dataflow.dflow INFO: Standard output for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/289d1aac-915b-44a3-807a-47b8a9c66ee5_calibrate_855138_50.stdout parsl.dataflow.dflow INFO: Standard error for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/289d1aac-915b-44a3-807a-47b8a9c66ee5_calibrate_855138_50.stderr parsl.dataflow.dflow INFO: Standard output for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/1eb8ae48-f248-4f25-9e2e-b35e9097220e_characterizeImage_855138_50.stdout parsl.dataflow.dflow INFO: Standard error for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/1eb8ae48-f248-4f25-9e2e-b35e9097220e_characterizeImage_855138_50.stderr parsl.dataflow.dflow INFO: Task 140 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9bb4a8152549d827e8829fa08061e6c with result from task 140 parsl.dataflow.dflow DEBUG: Task 140 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/4bc2e91d-3d07-45e8-8a78-a414bb2ce606_calibrate_855044_34.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/4bc2e91d-3d07-45e8-8a78-a414bb2ce606_calibrate_855044_34.stderr parsl.dataflow.dflow INFO: Task 590 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9816c4a87fb3bce7c9c3033a597d58da with result from task 590 parsl.dataflow.dflow DEBUG: Task 590 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/895e59ca-1c93-478a-af24-b564d7d46af6_calibrate_855042_48.stdout parsl.dataflow.dflow INFO: Standard error for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/895e59ca-1c93-478a-af24-b564d7d46af6_calibrate_855042_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:23 proc_lsst.multi:146 [INFO] found job 16912338 in provider checkpoint proc_lsst.multi INFO: found job 16912338 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 502 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2db874c7c92908b98d6010f3ac604a52 with result from task 502 parsl.dataflow.dflow DEBUG: Task 502 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 503 has memoization hash 13f57901fbdd0de46a524b9e60cc62ec parsl.dataflow.memoization INFO: Task 503 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456ce0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 503 try 0 launched on executor multi with executor id 997 parsl.dataflow.dflow INFO: Standard output for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/7fc0adcc-1cca-4bab-8788-cc19e86d2e95_calibrate_855042_13.stdout parsl.dataflow.dflow INFO: Standard error for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/7fc0adcc-1cca-4bab-8788-cc19e86d2e95_calibrate_855042_13.stderr parsl.dataflow.dflow INFO: Standard output for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/d51ca415-1542-4450-8109-7d6fc1664e4c_characterizeImage_855042_13.stdout parsl.dataflow.dflow INFO: Standard error for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/d51ca415-1542-4450-8109-7d6fc1664e4c_characterizeImage_855042_13.stderr parsl.dataflow.dflow INFO: Task 788 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0bc96c392a1d308488022913757653c4 with result from task 788 parsl.dataflow.dflow DEBUG: Task 788 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/2e15d384-6efe-48c7-861b-2c418282916e_calibrate_855137_50.stdout parsl.dataflow.dflow INFO: Standard error for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/2e15d384-6efe-48c7-861b-2c418282916e_calibrate_855137_50.stderr parsl.dataflow.dflow INFO: Task 806 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3eabfa1a273b30f42b4412cbafafeb11 with result from task 806 parsl.dataflow.dflow DEBUG: Task 806 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/fdb40be7-6ec1-4a81-9d30-02ec94e7091e_calibrate_855136_6.stdout parsl.dataflow.dflow INFO: Standard error for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/fdb40be7-6ec1-4a81-9d30-02ec94e7091e_calibrate_855136_6.stderr parsl.dataflow.dflow INFO: Task 413 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b1f6addcec4819fd9bc307768b931835 with result from task 413 parsl.dataflow.dflow DEBUG: Task 413 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/b273e99e-6cc4-4147-8538-e81cf13615db_calibrate_855136_38.stdout parsl.dataflow.dflow INFO: Standard error for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/b273e99e-6cc4-4147-8538-e81cf13615db_calibrate_855136_38.stderr parsl.dataflow.dflow INFO: Task 416 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7414d049fbe8eb9e63125ce5dde2a860 with result from task 416 parsl.dataflow.dflow DEBUG: Task 416 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/07ffbe8d-af9c-47e1-b701-8bc202ddaed9_calibrate_855042_12.stdout parsl.dataflow.dflow INFO: Standard error for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/07ffbe8d-af9c-47e1-b701-8bc202ddaed9_calibrate_855042_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:24 proc_lsst.multi:146 [INFO] found job 16912339 in provider checkpoint proc_lsst.multi INFO: found job 16912339 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 785 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef1fac06942ec68b9755d2bb29715e34 with result from task 785 parsl.dataflow.dflow DEBUG: Task 785 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/857bf01a-ebe1-4f66-9d8d-6a89b02bccb3_calibrate_855043_57.stdout parsl.dataflow.dflow INFO: Standard error for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/857bf01a-ebe1-4f66-9d8d-6a89b02bccb3_calibrate_855043_57.stderr parsl.dataflow.dflow INFO: Task 407 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a7d24710b49044cdd762e554e84c34fc with result from task 407 parsl.dataflow.dflow DEBUG: Task 407 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/2b45802e-5dd6-456d-83d0-eaf775bf7b50_calibrate_855137_45.stdout parsl.dataflow.dflow INFO: Standard error for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/2b45802e-5dd6-456d-83d0-eaf775bf7b50_calibrate_855137_45.stderr parsl.dataflow.dflow INFO: Task 335 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2965993b3cf1a4d6bcb19ea48031737 with result from task 335 parsl.dataflow.dflow DEBUG: Task 335 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/f3af6665-f039-4ca8-b874-eb372b9ba647_calibrate_855138_24.stdout parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/f3af6665-f039-4ca8-b874-eb372b9ba647_calibrate_855138_24.stderr parsl.dataflow.dflow INFO: Task 188 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e636d121812330938bff590699c7b3b7 with result from task 188 parsl.dataflow.dflow DEBUG: Task 188 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/ef1d4320-4f93-4fbe-b05e-d81ab13cadef_calibrate_855044_44.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/ef1d4320-4f93-4fbe-b05e-d81ab13cadef_calibrate_855044_44.stderr parsl.dataflow.dflow INFO: Task 457 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 073df0b1db7be7d527ffc19685f3179b with result from task 457 parsl.dataflow.dflow DEBUG: Task 457 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 458 has memoization hash 266728b56fa93883b920a09d36037d87 parsl.dataflow.memoization INFO: Task 458 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456cc180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 458 try 0 launched on executor multi with executor id 998 parsl.dataflow.dflow INFO: Standard output for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/14501953-964d-4519-b0dd-e6d139c446ea_calibrate_855136_4.stdout parsl.dataflow.dflow INFO: Standard error for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/14501953-964d-4519-b0dd-e6d139c446ea_calibrate_855136_4.stderr parsl.dataflow.dflow INFO: Standard output for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/85300276-6776-4408-9911-8901979e9ca1_characterizeImage_855136_4.stdout parsl.dataflow.dflow INFO: Standard error for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/85300276-6776-4408-9911-8901979e9ca1_characterizeImage_855136_4.stderr parsl.dataflow.dflow INFO: Task 239 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 738a8e569e15ef67737e6e019143c5a6 with result from task 239 parsl.dataflow.dflow DEBUG: Task 239 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/9f1494fd-6b37-459a-a5f7-061654f8b538_calibrate_855138_4.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/9f1494fd-6b37-459a-a5f7-061654f8b538_calibrate_855138_4.stderr parsl.dataflow.dflow INFO: Task 755 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f573a3a4ccbf03a8a17e6d1f1fda38fe with result from task 755 parsl.dataflow.dflow DEBUG: Task 755 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/5a535596-b295-4a95-9f09-7143dd685eea_calibrate_855042_47.stdout parsl.dataflow.dflow INFO: Standard error for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/5a535596-b295-4a95-9f09-7143dd685eea_calibrate_855042_47.stderr parsl.dataflow.dflow INFO: Task 245 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2121a71fb26ddaa807e4da7f4a088a52 with result from task 245 parsl.dataflow.dflow DEBUG: Task 245 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/5d6661e9-5552-4b7b-9830-ce2b5bcdda3f_calibrate_855043_13.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/5d6661e9-5552-4b7b-9830-ce2b5bcdda3f_calibrate_855043_13.stderr parsl.dataflow.dflow INFO: Task 781 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be255e7f575655d67c58982ac04175e6 with result from task 781 parsl.dataflow.dflow DEBUG: Task 781 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 782 has memoization hash d57289390e0dc16e80814c56fa5c34c0 parsl.dataflow.memoization INFO: Task 782 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 782 try 0 launched on executor multi with executor id 999 parsl.dataflow.dflow INFO: Standard output for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/2dd59477-04b8-477c-b030-24598cbfdfbb_calibrate_855042_22.stdout parsl.dataflow.dflow INFO: Standard error for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/2dd59477-04b8-477c-b030-24598cbfdfbb_calibrate_855042_22.stderr parsl.dataflow.dflow INFO: Standard output for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/9aff9434-8a09-4f9d-9841-4e024676275f_characterizeImage_855042_22.stdout parsl.dataflow.dflow INFO: Standard error for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/9aff9434-8a09-4f9d-9841-4e024676275f_characterizeImage_855042_22.stderr parsl.dataflow.dflow INFO: Task 928 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2557b1b50f460699909e9c0ae795f2ff with result from task 928 parsl.dataflow.dflow DEBUG: Task 928 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 929 has memoization hash a4fc6ddcde3a65955fbd1fc3750adfcc parsl.dataflow.memoization INFO: Task 929 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4549ad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 929 try 0 launched on executor multi with executor id 1000 parsl.dataflow.dflow INFO: Standard output for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/1568681b-39f8-48d7-adc8-183389975fbd_calibrate_855138_25.stdout parsl.dataflow.dflow INFO: Standard error for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/1568681b-39f8-48d7-adc8-183389975fbd_calibrate_855138_25.stderr parsl.dataflow.dflow INFO: Standard output for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/67ed7dfe-c6b6-40b4-9d69-3a7b76480482_characterizeImage_855138_25.stdout parsl.dataflow.dflow INFO: Standard error for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/67ed7dfe-c6b6-40b4-9d69-3a7b76480482_characterizeImage_855138_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:26 proc_lsst.multi:146 [INFO] found job 16912340 in provider checkpoint proc_lsst.multi INFO: found job 16912340 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 1087 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7e0f0b141e83fa911f3bdc84cf68d42 with result from task 1087 parsl.dataflow.dflow DEBUG: Task 1087 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1088 has memoization hash d0d89a0cacd4056b7a5a4fa6cd6684ce parsl.dataflow.memoization INFO: Task 1088 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45390b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1088 try 0 launched on executor multi with executor id 1001 parsl.dataflow.dflow INFO: Standard output for task 1088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/e07dd087-a68f-41e1-b4ff-dbb7cd785fed_calibrate_855136_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/e07dd087-a68f-41e1-b4ff-dbb7cd785fed_calibrate_855136_56.stderr parsl.dataflow.dflow INFO: Standard output for task 1087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/4602f08e-15b0-4ced-abf2-7a93e0005b5a_characterizeImage_855136_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/4602f08e-15b0-4ced-abf2-7a93e0005b5a_characterizeImage_855136_56.stderr parsl.dataflow.dflow INFO: Task 221 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad8d5c449595bc19262aa3a3de050a1a with result from task 221 parsl.dataflow.dflow DEBUG: Task 221 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/911d9d41-4815-4487-afa8-d011c9437ab0_calibrate_855043_32.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/911d9d41-4815-4487-afa8-d011c9437ab0_calibrate_855043_32.stderr parsl.dataflow.dflow INFO: Task 830 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4838ef3382ade9a5a20fa840212591dd with result from task 830 parsl.dataflow.dflow DEBUG: Task 830 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/7e793946-2771-460f-8273-72aeb250275d_calibrate_855136_5.stdout parsl.dataflow.dflow INFO: Standard error for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/7e793946-2771-460f-8273-72aeb250275d_calibrate_855136_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:27 proc_lsst.multi:146 [INFO] found job 16912341 in provider checkpoint proc_lsst.multi INFO: found job 16912341 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 979 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 19c4d6cfdedaa3325757b968648587d0 with result from task 979 parsl.dataflow.dflow DEBUG: Task 979 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 980 has memoization hash 824607e4459e64ce7b387656e8f1b727 parsl.dataflow.memoization INFO: Task 980 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454f93a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 980 try 0 launched on executor multi with executor id 1002 parsl.dataflow.dflow INFO: Standard output for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/fb556dc9-a9f2-4c0b-a2cb-c567076d9abd_calibrate_855138_11.stdout parsl.dataflow.dflow INFO: Standard error for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/fb556dc9-a9f2-4c0b-a2cb-c567076d9abd_calibrate_855138_11.stderr parsl.dataflow.dflow INFO: Standard output for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/18b786a3-284b-4df4-a757-01fc90707683_characterizeImage_855138_11.stdout parsl.dataflow.dflow INFO: Standard error for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/18b786a3-284b-4df4-a757-01fc90707683_characterizeImage_855138_11.stderr parsl.dataflow.dflow INFO: Task 439 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f98d9e78d3498b9222f410447cb2f0d with result from task 439 parsl.dataflow.dflow DEBUG: Task 439 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 440 has memoization hash f96b292e90451f8e11b1a0a2b56ee9ef parsl.dataflow.memoization INFO: Task 440 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a6200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 440 try 0 launched on executor multi with executor id 1003 parsl.dataflow.dflow INFO: Standard output for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/301130dd-5229-4eaa-9666-c21f9ce7ad3c_calibrate_855044_23.stdout parsl.dataflow.dflow INFO: Standard error for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/301130dd-5229-4eaa-9666-c21f9ce7ad3c_calibrate_855044_23.stderr parsl.dataflow.dflow INFO: Standard output for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/2edc0738-eb77-40ff-a934-6a8b6bf2b91d_characterizeImage_855044_23.stdout parsl.dataflow.dflow INFO: Standard error for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/2edc0738-eb77-40ff-a934-6a8b6bf2b91d_characterizeImage_855044_23.stderr parsl.dataflow.dflow INFO: Task 1108 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e9046dab0c5bada3e9b75b83b91ce97 with result from task 1108 parsl.dataflow.dflow DEBUG: Task 1108 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1109 has memoization hash c2092c6e8379eb6e39f1b3d4c24033ca parsl.dataflow.memoization INFO: Task 1109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa453927a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1109 try 0 launched on executor multi with executor id 1004 parsl.dataflow.dflow INFO: Standard output for task 1109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/d74e48bc-a455-4453-877c-7b8075c74c09_calibrate_855138_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/d74e48bc-a455-4453-877c-7b8075c74c09_calibrate_855138_15.stderr parsl.dataflow.dflow INFO: Standard output for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/b9849438-8cfc-4435-a478-627529b331f8_characterizeImage_855138_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/b9849438-8cfc-4435-a478-627529b331f8_characterizeImage_855138_15.stderr parsl.dataflow.dflow INFO: Task 557 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c31ac87fc83367453aa85371e57f1ef2 with result from task 557 parsl.dataflow.dflow DEBUG: Task 557 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/bf6eec03-efc2-4760-90ef-a884f8cf2fb0_calibrate_855042_1.stdout parsl.dataflow.dflow INFO: Standard error for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/bf6eec03-efc2-4760-90ef-a884f8cf2fb0_calibrate_855042_1.stderr parsl.dataflow.dflow INFO: Task 1052 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b57fdee9c16ad7465a730e15a5dd77d with result from task 1052 parsl.dataflow.dflow DEBUG: Task 1052 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/a6ba4420-61de-4182-b505-a8a2996a099b_calibrate_855043_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/a6ba4420-61de-4182-b505-a8a2996a099b_calibrate_855043_51.stderr parsl.dataflow.dflow INFO: Task 634 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3646231171e0ca43a6f5ab7ca1c0b42 with result from task 634 parsl.dataflow.dflow DEBUG: Task 634 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 635 has memoization hash 1af496d4449423c53b4c0c34b93697be parsl.dataflow.memoization INFO: Task 635 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45743ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 635 try 0 launched on executor multi with executor id 1005 parsl.dataflow.dflow INFO: Standard output for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/9fb62733-f011-4bb0-a94d-301139db1405_calibrate_855044_37.stdout parsl.dataflow.dflow INFO: Standard error for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/9fb62733-f011-4bb0-a94d-301139db1405_calibrate_855044_37.stderr parsl.dataflow.dflow INFO: Standard output for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/193fd599-2bd6-42f8-8431-c9e95745e8a1_characterizeImage_855044_37.stdout parsl.dataflow.dflow INFO: Standard error for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/193fd599-2bd6-42f8-8431-c9e95745e8a1_characterizeImage_855044_37.stderr parsl.dataflow.dflow INFO: Task 95 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a18a11de59bdc7edecab58dd665c45ca with result from task 95 parsl.dataflow.dflow DEBUG: Task 95 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/0bf9594c-4601-4c04-bef1-825fe56383d4_calibrate_855137_30.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/0bf9594c-4601-4c04-bef1-825fe56383d4_calibrate_855137_30.stderr parsl.dataflow.dflow INFO: Task 104 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66f28f51e725d52a76595e8799b8c87d with result from task 104 parsl.dataflow.dflow DEBUG: Task 104 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/5cafa3b6-95b5-42da-b58f-4c4afc019fbd_calibrate_855136_3.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/5cafa3b6-95b5-42da-b58f-4c4afc019fbd_calibrate_855136_3.stderr parsl.dataflow.dflow INFO: Task 149 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a80bb716a65a1202d466c5bdf3d20a1 with result from task 149 parsl.dataflow.dflow DEBUG: Task 149 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/775b497a-1bce-4ca5-8d2e-73cfe8735ee8_calibrate_855138_37.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/775b497a-1bce-4ca5-8d2e-73cfe8735ee8_calibrate_855138_37.stderr parsl.dataflow.dflow INFO: Task 914 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4441f72f3af1cf62f4b6f570965b4a63 with result from task 914 parsl.dataflow.dflow DEBUG: Task 914 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/12dbde3a-504a-4689-9af1-4e696ba5deb0_calibrate_855042_24.stdout parsl.dataflow.dflow INFO: Standard error for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/12dbde3a-504a-4689-9af1-4e696ba5deb0_calibrate_855042_24.stderr parsl.dataflow.dflow INFO: Task 143 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6409f1f5088a129723b392b9d2c2c19e with result from task 143 parsl.dataflow.dflow DEBUG: Task 143 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/41fe2494-97be-403b-8f2c-119d7d1c31f9_calibrate_855044_24.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/41fe2494-97be-403b-8f2c-119d7d1c31f9_calibrate_855044_24.stderr parsl.dataflow.dflow INFO: Task 284 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a86ef2bd71b7ba61f90ffef903aa9e1 with result from task 284 parsl.dataflow.dflow DEBUG: Task 284 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/724cbfce-0c03-4f11-9b40-f7d2cb1e45e6_calibrate_855043_36.stdout parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/724cbfce-0c03-4f11-9b40-f7d2cb1e45e6_calibrate_855043_36.stderr parsl.dataflow.dflow INFO: Task 833 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b008462d3be20429d4fb8bcb7138b6f4 with result from task 833 parsl.dataflow.dflow DEBUG: Task 833 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/8368ef91-bc08-4ea6-852a-b0e95cf63ab2_calibrate_855043_34.stdout parsl.dataflow.dflow INFO: Standard error for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/8368ef91-bc08-4ea6-852a-b0e95cf63ab2_calibrate_855043_34.stderr parsl.dataflow.dflow INFO: Task 1033 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 202b930ffca9ee2d7015182856d3f74c with result from task 1033 parsl.dataflow.dflow DEBUG: Task 1033 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1034 has memoization hash 1e008a7936b9cb992940cb608f3b1f66 parsl.dataflow.memoization INFO: Task 1034 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45541a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1034 try 0 launched on executor multi with executor id 1006 parsl.dataflow.dflow INFO: Standard output for task 1034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/3439bc5e-a29d-4a65-ba28-6f3fe022e3d2_calibrate_855043_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/3439bc5e-a29d-4a65-ba28-6f3fe022e3d2_calibrate_855043_38.stderr parsl.dataflow.dflow INFO: Standard output for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/e8148b0d-7a16-4dba-8b12-b7f19f4d28b7_characterizeImage_855043_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/e8148b0d-7a16-4dba-8b12-b7f19f4d28b7_characterizeImage_855043_38.stderr parsl.dataflow.dflow INFO: Task 818 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 90d7597e9ed2b3ceb6db7410eb1924be with result from task 818 parsl.dataflow.dflow DEBUG: Task 818 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/93b7738f-b669-4b28-8119-96f0a5348424_calibrate_855043_35.stdout parsl.dataflow.dflow INFO: Standard error for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/93b7738f-b669-4b28-8119-96f0a5348424_calibrate_855043_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:29 proc_lsst.multi:146 [INFO] found job 16912342 in provider checkpoint proc_lsst.multi INFO: found job 16912342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:29 proc_lsst.multi:146 [INFO] found job 16912343 in provider checkpoint proc_lsst.multi INFO: found job 16912343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 382 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e77257b064c2824f4a646aa40bf7f4b with result from task 382 parsl.dataflow.dflow DEBUG: Task 382 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 383 has memoization hash 9a50a98f7fdfb6c700977af186f5564d parsl.dataflow.memoization INFO: Task 383 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4687ad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 383 try 0 launched on executor multi with executor id 1007 parsl.dataflow.dflow INFO: Standard output for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/109a6ba6-944d-4653-84a9-80eb36e49663_calibrate_855042_59.stdout parsl.dataflow.dflow INFO: Standard error for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/109a6ba6-944d-4653-84a9-80eb36e49663_calibrate_855042_59.stderr parsl.dataflow.dflow INFO: Standard output for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/548a6c7f-2d56-4e95-98da-ff264dbc9124_characterizeImage_855042_59.stdout parsl.dataflow.dflow INFO: Standard error for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/548a6c7f-2d56-4e95-98da-ff264dbc9124_characterizeImage_855042_59.stderr parsl.dataflow.dflow INFO: Task 836 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry efb623ddabc49bd46677215eb10c94a0 with result from task 836 parsl.dataflow.dflow DEBUG: Task 836 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/6163dcab-8efd-4efa-a544-44adbbe7fb6a_calibrate_855044_42.stdout parsl.dataflow.dflow INFO: Standard error for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/6163dcab-8efd-4efa-a544-44adbbe7fb6a_calibrate_855044_42.stderr parsl.dataflow.dflow INFO: Task 779 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 20ac51adba9114c515b77b4469572ef6 with result from task 779 parsl.dataflow.dflow DEBUG: Task 779 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/64e102e7-38f2-4033-b50b-4ee1cb0c546c_calibrate_855044_26.stdout parsl.dataflow.dflow INFO: Standard error for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/64e102e7-38f2-4033-b50b-4ee1cb0c546c_calibrate_855044_26.stderr parsl.dataflow.dflow INFO: Task 872 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 90ee696a558599d1d83929c352cefc4c with result from task 872 parsl.dataflow.dflow DEBUG: Task 872 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/0644fab4-4c76-4339-b03a-6deaaa85e837_calibrate_855043_20.stdout parsl.dataflow.dflow INFO: Standard error for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/0644fab4-4c76-4339-b03a-6deaaa85e837_calibrate_855043_20.stderr parsl.dataflow.dflow INFO: Task 50 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7592c2c0ce9e19cc2d5f7d2e92de1b4c with result from task 50 parsl.dataflow.dflow DEBUG: Task 50 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/f3bf14dd-0f16-46f6-bfe4-a02c71f2d141_calibrate_855138_55.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/f3bf14dd-0f16-46f6-bfe4-a02c71f2d141_calibrate_855138_55.stderr parsl.dataflow.dflow INFO: Task 272 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7b12b0725cd0abe9a3f3515d1887e18 with result from task 272 parsl.dataflow.dflow DEBUG: Task 272 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/ab5b17b7-2f19-4c9f-9775-1f774a10b55c_calibrate_855042_16.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/ab5b17b7-2f19-4c9f-9775-1f774a10b55c_calibrate_855042_16.stderr parsl.dataflow.dflow INFO: Task 107 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4bc3ca2c2a209336ccadfb0e3b31044d with result from task 107 parsl.dataflow.dflow DEBUG: Task 107 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/c7c4fda7-e1e4-4bf0-87a7-022515dc1cb4_calibrate_855042_14.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/c7c4fda7-e1e4-4bf0-87a7-022515dc1cb4_calibrate_855042_14.stderr parsl.dataflow.dflow INFO: Task 460 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ab830f01f2c092280829c18d34b44b3 with result from task 460 parsl.dataflow.dflow DEBUG: Task 460 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 461 has memoization hash 28be2d6b73045006503500af207cc7c9 parsl.dataflow.memoization INFO: Task 461 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456cc540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 461 try 0 launched on executor multi with executor id 1008 parsl.dataflow.dflow INFO: Standard output for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/fd781cf1-a7ca-4472-b502-31932bf3d056_calibrate_855042_19.stdout parsl.dataflow.dflow INFO: Standard error for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/fd781cf1-a7ca-4472-b502-31932bf3d056_calibrate_855042_19.stderr parsl.dataflow.dflow INFO: Standard output for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/6fc0e6da-cbce-4330-8098-d3a3922aa38e_characterizeImage_855042_19.stdout parsl.dataflow.dflow INFO: Standard error for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/6fc0e6da-cbce-4330-8098-d3a3922aa38e_characterizeImage_855042_19.stderr parsl.dataflow.dflow INFO: Task 197 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 583179ca727b52a599aa4cb6bb4e4ec4 with result from task 197 parsl.dataflow.dflow DEBUG: Task 197 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/2c28dbba-988a-44ca-8394-f21d0bf20496_calibrate_855136_45.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/2c28dbba-988a-44ca-8394-f21d0bf20496_calibrate_855136_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:31 proc_lsst.multi:146 [INFO] found job 16912344 in provider checkpoint proc_lsst.multi INFO: found job 16912344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:32 proc_lsst.multi:146 [INFO] found job 16912345 in provider checkpoint proc_lsst.multi INFO: found job 16912345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 89 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3beddf237fc91b5aed6f862f2766589 with result from task 89 parsl.dataflow.dflow DEBUG: Task 89 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/ea463d49-9146-4855-a3ef-eca4ba2f395a_calibrate_855044_28.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/ea463d49-9146-4855-a3ef-eca4ba2f395a_calibrate_855044_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:32 proc_lsst.multi:146 [INFO] found job 16912346 in provider checkpoint proc_lsst.multi INFO: found job 16912346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 26 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b4298426886fc77f48bf7ed862bfbda with result from task 26 parsl.dataflow.dflow DEBUG: Task 26 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/49200616-d1b0-4a76-a832-e9f87d46f0c4_calibrate_855044_22.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/49200616-d1b0-4a76-a832-e9f87d46f0c4_calibrate_855044_22.stderr parsl.dataflow.dflow INFO: Task 365 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e94c2e69b7b53c7e9da18d62c443923d with result from task 365 parsl.dataflow.dflow DEBUG: Task 365 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/daa79955-7230-40e2-b1b4-89f0125397fb_calibrate_855043_30.stdout parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/daa79955-7230-40e2-b1b4-89f0125397fb_calibrate_855043_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:32 proc_lsst.multi:146 [INFO] found job 16912347 in provider checkpoint proc_lsst.multi INFO: found job 16912347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:33 proc_lsst.multi:146 [INFO] found job 16912348 in provider checkpoint proc_lsst.multi INFO: found job 16912348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 923 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d083ab974a92d68576ff12c39e1e523 with result from task 923 parsl.dataflow.dflow DEBUG: Task 923 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/915e5395-6835-4105-899f-fd2cba696be1_calibrate_855044_55.stdout parsl.dataflow.dflow INFO: Standard error for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/915e5395-6835-4105-899f-fd2cba696be1_calibrate_855044_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:33 proc_lsst.multi:146 [INFO] found job 16912349 in provider checkpoint proc_lsst.multi INFO: found job 16912349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 167 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b960b637d560c364e3f13a94477db22 with result from task 167 parsl.dataflow.dflow DEBUG: Task 167 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/9bc30c84-9093-4d8e-b8da-57b7523b8f0e_calibrate_855042_20.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/9bc30c84-9093-4d8e-b8da-57b7523b8f0e_calibrate_855042_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:33 proc_lsst.multi:146 [INFO] found job 16912350 in provider checkpoint proc_lsst.multi INFO: found job 16912350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 260 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b53bb93043ee931943088df609d31bc with result from task 260 parsl.dataflow.dflow DEBUG: Task 260 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/6d64b5ce-7116-4855-881c-c7588f582d9b_calibrate_855137_5.stdout parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/6d64b5ce-7116-4855-881c-c7588f582d9b_calibrate_855137_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:34 proc_lsst.multi:146 [INFO] found job 16912351 in provider checkpoint proc_lsst.multi INFO: found job 16912351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 206 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ffb6fc8aa75d88dc43055b93f37098b with result from task 206 parsl.dataflow.dflow DEBUG: Task 206 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/ea3d9cf7-0428-4761-a66f-c3c19a67e871_calibrate_855044_33.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/ea3d9cf7-0428-4761-a66f-c3c19a67e871_calibrate_855044_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:34 proc_lsst.multi:146 [INFO] found job 16912352 in provider checkpoint proc_lsst.multi INFO: found job 16912352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 68 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63ac3e0c3f30e659a98f949f441a2e03 with result from task 68 parsl.dataflow.dflow DEBUG: Task 68 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/732d9894-9437-4b01-8483-feca2e0a6e9a_calibrate_855136_34.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/732d9894-9437-4b01-8483-feca2e0a6e9a_calibrate_855136_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:34 proc_lsst.multi:146 [INFO] found job 16912353 in provider checkpoint proc_lsst.multi INFO: found job 16912353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:34 proc_lsst.multi:146 [INFO] found job 16912354 in provider checkpoint proc_lsst.multi INFO: found job 16912354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:35 proc_lsst.multi:146 [INFO] found job 16912355 in provider checkpoint proc_lsst.multi INFO: found job 16912355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:35 proc_lsst.multi:146 [INFO] found job 16912356 in provider checkpoint proc_lsst.multi INFO: found job 16912356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 266 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24cc5243d6d479c062ffdfe65b249a9a with result from task 266 parsl.dataflow.dflow DEBUG: Task 266 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/b82ce910-d779-489e-be01-31f7f39d10b5_calibrate_855044_40.stdout parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/b82ce910-d779-489e-be01-31f7f39d10b5_calibrate_855044_40.stderr parsl.dataflow.dflow INFO: Task 53 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5afdb029cd75ed55e21eb7439d8e7a44 with result from task 53 parsl.dataflow.dflow DEBUG: Task 53 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/11315694-7bcf-4a5e-87b5-c6d83320318a_calibrate_855137_34.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/11315694-7bcf-4a5e-87b5-c6d83320318a_calibrate_855137_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:35 proc_lsst.multi:146 [INFO] found job 16912359 in provider checkpoint proc_lsst.multi INFO: found job 16912359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 908 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4659a6275f3cb817d88f044188f315b with result from task 908 parsl.dataflow.dflow DEBUG: Task 908 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/74c3fb69-5670-478a-a471-2137a9d7dec2_calibrate_855042_8.stdout parsl.dataflow.dflow INFO: Standard error for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/74c3fb69-5670-478a-a471-2137a9d7dec2_calibrate_855042_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:36 proc_lsst.multi:146 [INFO] found job 16912360 in provider checkpoint proc_lsst.multi INFO: found job 16912360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:36 proc_lsst.multi:146 [INFO] found job 16912361 in provider checkpoint proc_lsst.multi INFO: found job 16912361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 125 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b4fb1b0c14b0d264612348235235541 with result from task 125 parsl.dataflow.dflow DEBUG: Task 125 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/ea18e29b-cd67-4cc7-b9fe-3c4a5fd6b746_calibrate_855137_51.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/ea18e29b-cd67-4cc7-b9fe-3c4a5fd6b746_calibrate_855137_51.stderr parsl.dataflow.dflow INFO: Task 578 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e66533046e073da9bf547d29af23d9d3 with result from task 578 parsl.dataflow.dflow DEBUG: Task 578 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/57a4c4c2-98fc-4c5f-b84f-9023d1a3730a_calibrate_855044_18.stdout parsl.dataflow.dflow INFO: Standard error for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/57a4c4c2-98fc-4c5f-b84f-9023d1a3730a_calibrate_855044_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:36 proc_lsst.multi:146 [INFO] found job 16912362 in provider checkpoint proc_lsst.multi INFO: found job 16912362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:37 proc_lsst.multi:146 [INFO] found job 16912363 in provider checkpoint proc_lsst.multi INFO: found job 16912363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 1057 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8ad216e439e0fbe646e3ce7aae924ee with result from task 1057 parsl.dataflow.dflow DEBUG: Task 1057 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1058 has memoization hash e8bd3ea3eff9cb81bc1db67b4e8efb4d parsl.dataflow.memoization INFO: Task 1058 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45542ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1058 try 0 launched on executor multi with executor id 1009 parsl.dataflow.dflow INFO: Standard output for task 1058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/459592e7-0a36-4b6c-aad1-2372d8ea0bd7_calibrate_855042_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/459592e7-0a36-4b6c-aad1-2372d8ea0bd7_calibrate_855042_37.stderr parsl.dataflow.dflow INFO: Standard output for task 1057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/9485a19e-1a02-4359-8cc1-c41bfdaa1117_characterizeImage_855042_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/9485a19e-1a02-4359-8cc1-c41bfdaa1117_characterizeImage_855042_37.stderr parsl.dataflow.dflow INFO: Task 137 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 20f1e0e3d87e38d724ef51c852d9ce70 with result from task 137 parsl.dataflow.dflow DEBUG: Task 137 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/d9b491ff-4776-40fc-ad2b-8c03e258ca30_calibrate_855137_57.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/d9b491ff-4776-40fc-ad2b-8c03e258ca30_calibrate_855137_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:37 proc_lsst.multi:146 [INFO] found job 16912364 in provider checkpoint proc_lsst.multi INFO: found job 16912364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 704 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81842c535ff8a638920773e56454da36 with result from task 704 parsl.dataflow.dflow DEBUG: Task 704 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/1a168c20-3169-498b-bda7-1ac0f54b172c_calibrate_855137_46.stdout parsl.dataflow.dflow INFO: Standard error for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/1a168c20-3169-498b-bda7-1ac0f54b172c_calibrate_855137_46.stderr parsl.dataflow.dflow INFO: Task 992 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b20541a6347cdc28fb1e83c8106fa28 with result from task 992 parsl.dataflow.dflow DEBUG: Task 992 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/9e9ec239-02ec-4d81-8a34-51e5eb27744e_calibrate_855044_11.stdout parsl.dataflow.dflow INFO: Standard error for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/9e9ec239-02ec-4d81-8a34-51e5eb27744e_calibrate_855044_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:37 proc_lsst.multi:146 [INFO] found job 16912365 in provider checkpoint proc_lsst.multi INFO: found job 16912365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 152 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1bbc24de45724b797f7e7fcb6c57a798 with result from task 152 parsl.dataflow.dflow DEBUG: Task 152 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/2c0a2bf8-00d4-4ac7-947b-1b17da00833b_calibrate_855043_29.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/2c0a2bf8-00d4-4ac7-947b-1b17da00833b_calibrate_855043_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:38 proc_lsst.multi:146 [INFO] found job 16912366 in provider checkpoint proc_lsst.multi INFO: found job 16912366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:38 proc_lsst.multi:146 [INFO] found job 16912367 in provider checkpoint proc_lsst.multi INFO: found job 16912367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 422 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da6f96335b4f9395d27295712b09576b with result from task 422 parsl.dataflow.dflow DEBUG: Task 422 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/abc8421d-0aab-44ec-be9c-5b8094e094cf_calibrate_855136_28.stdout parsl.dataflow.dflow INFO: Standard error for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/abc8421d-0aab-44ec-be9c-5b8094e094cf_calibrate_855136_28.stderr parsl.dataflow.dflow INFO: Task 233 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5322c1df78d38b4c73830c9a3ba72e64 with result from task 233 parsl.dataflow.dflow DEBUG: Task 233 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/0e5cf54e-cc25-49b7-9538-d3077983f08c_calibrate_855044_48.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/0e5cf54e-cc25-49b7-9538-d3077983f08c_calibrate_855044_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:38 proc_lsst.multi:146 [INFO] found job 16912368 in provider checkpoint proc_lsst.multi INFO: found job 16912368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:38 proc_lsst.multi:146 [INFO] found job 16912369 in provider checkpoint proc_lsst.multi INFO: found job 16912369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 604 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c80611ddf11d35d8d727ea025bb9136 with result from task 604 parsl.dataflow.dflow DEBUG: Task 604 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 605 has memoization hash 71407d2e25d11604c551043314f8ac1a parsl.dataflow.memoization INFO: Task 605 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45740e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 605 try 0 launched on executor multi with executor id 1010 parsl.dataflow.dflow INFO: Standard output for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/61cc534e-765c-40d3-8942-1635c2506c09_calibrate_855137_23.stdout parsl.dataflow.dflow INFO: Standard error for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/61cc534e-765c-40d3-8942-1635c2506c09_calibrate_855137_23.stderr parsl.dataflow.dflow INFO: Standard output for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/2bcbb6e2-c6c2-43c1-b057-c8834197bea0_characterizeImage_855137_23.stdout parsl.dataflow.dflow INFO: Standard error for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/2bcbb6e2-c6c2-43c1-b057-c8834197bea0_characterizeImage_855137_23.stderr parsl.dataflow.dflow INFO: Task 695 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d800cf713cda3b3ebd5058993029cc6 with result from task 695 parsl.dataflow.dflow DEBUG: Task 695 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/437ab5fd-1bac-40bd-acc6-ca0ed6d657a3_calibrate_855138_29.stdout parsl.dataflow.dflow INFO: Standard error for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/437ab5fd-1bac-40bd-acc6-ca0ed6d657a3_calibrate_855138_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:39 proc_lsst.multi:146 [INFO] found job 16912370 in provider checkpoint proc_lsst.multi INFO: found job 16912370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:39 proc_lsst.multi:146 [INFO] found job 16912371 in provider checkpoint proc_lsst.multi INFO: found job 16912371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 161 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ace94b5c75f7c6420e728e845fe9183 with result from task 161 parsl.dataflow.dflow DEBUG: Task 161 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/2a1a9977-1b21-463a-b47e-a43a961a36b8_calibrate_855138_57.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/2a1a9977-1b21-463a-b47e-a43a961a36b8_calibrate_855138_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:39 proc_lsst.multi:146 [INFO] found job 16912372 in provider checkpoint proc_lsst.multi INFO: found job 16912372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 242 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e06b8afde1b1d431728f48599c937304 with result from task 242 parsl.dataflow.dflow DEBUG: Task 242 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/a13abcfe-b214-4dbb-94bc-a5aaad76570b_calibrate_855136_8.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/a13abcfe-b214-4dbb-94bc-a5aaad76570b_calibrate_855136_8.stderr parsl.dataflow.dflow INFO: Task 550 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e41793b2157dedef250dfd093d9e825 with result from task 550 parsl.dataflow.dflow DEBUG: Task 550 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 551 has memoization hash 0883b878762747c05543d28acdda7e2d parsl.dataflow.memoization INFO: Task 551 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45701c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 551 try 0 launched on executor multi with executor id 1011 parsl.dataflow.dflow INFO: Standard output for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/77694236-45c0-4561-a51c-341c1f724b90_calibrate_855044_62.stdout parsl.dataflow.dflow INFO: Standard error for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/77694236-45c0-4561-a51c-341c1f724b90_calibrate_855044_62.stderr parsl.dataflow.dflow INFO: Standard output for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/de577080-318d-4964-bbec-e20e0cd52651_characterizeImage_855044_62.stdout parsl.dataflow.dflow INFO: Standard error for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/de577080-318d-4964-bbec-e20e0cd52651_characterizeImage_855044_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:40 proc_lsst.multi:146 [INFO] found job 16912373 in provider checkpoint proc_lsst.multi INFO: found job 16912373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 352 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3ecd7a9fb332051b486abcd58e8a5c5 with result from task 352 parsl.dataflow.dflow DEBUG: Task 352 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 353 has memoization hash 032fef6755238c1c697e50569749f3a3 parsl.dataflow.memoization INFO: Task 353 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46879260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 353 try 0 launched on executor multi with executor id 1012 parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/08ad799a-07fa-46ae-b322-64792da32545_calibrate_855042_38.stdout parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/08ad799a-07fa-46ae-b322-64792da32545_calibrate_855042_38.stderr parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/a7fdd3dd-173a-4bc0-b2b6-d9c8436cabe9_characterizeImage_855042_38.stdout parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/a7fdd3dd-173a-4bc0-b2b6-d9c8436cabe9_characterizeImage_855042_38.stderr parsl.dataflow.dflow INFO: Task 572 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c747fec1799467ebe099e7b944183ff with result from task 572 parsl.dataflow.dflow DEBUG: Task 572 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/9361bbd5-c16c-42f8-bd22-98c4bfce5474_calibrate_855043_53.stdout parsl.dataflow.dflow INFO: Standard error for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/9361bbd5-c16c-42f8-bd22-98c4bfce5474_calibrate_855043_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:40 proc_lsst.multi:146 [INFO] found job 16912374 in provider checkpoint proc_lsst.multi INFO: found job 16912374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 425 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a13da714a6cff212721e31a993830818 with result from task 425 parsl.dataflow.dflow DEBUG: Task 425 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/d805cbfe-f750-43fb-8251-d515f2adf745_calibrate_855044_51.stdout parsl.dataflow.dflow INFO: Standard error for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/d805cbfe-f750-43fb-8251-d515f2adf745_calibrate_855044_51.stderr parsl.dataflow.dflow INFO: Task 224 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1a200adad3a8a7e66fd101c3f4f5e5b with result from task 224 parsl.dataflow.dflow DEBUG: Task 224 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/f44d25b0-27ae-46eb-ad72-97d2639831ca_calibrate_855044_32.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/f44d25b0-27ae-46eb-ad72-97d2639831ca_calibrate_855044_32.stderr parsl.dataflow.dflow INFO: Task 800 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83265f3d5bba8b78c2a470334aa9cba0 with result from task 800 parsl.dataflow.dflow DEBUG: Task 800 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/c35c0c7b-9cd0-4e6e-aca7-4951b1282f2f_calibrate_855138_35.stdout parsl.dataflow.dflow INFO: Standard error for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/c35c0c7b-9cd0-4e6e-aca7-4951b1282f2f_calibrate_855138_35.stderr parsl.dataflow.dflow INFO: Task 899 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 555a0eabc9ecdcbe33892af8f174c107 with result from task 899 parsl.dataflow.dflow DEBUG: Task 899 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/8af92a7f-f077-460b-80fb-4646cddd3c0e_calibrate_855136_52.stdout parsl.dataflow.dflow INFO: Standard error for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/8af92a7f-f077-460b-80fb-4646cddd3c0e_calibrate_855136_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:40 proc_lsst.multi:146 [INFO] found job 16912375 in provider checkpoint proc_lsst.multi INFO: found job 16912375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 665 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d79729f9328b99e1dde13b790910abd5 with result from task 665 parsl.dataflow.dflow DEBUG: Task 665 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/670d6def-4de8-4de7-b085-b2fd8758aa18_calibrate_855042_18.stdout parsl.dataflow.dflow INFO: Standard error for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/670d6def-4de8-4de7-b085-b2fd8758aa18_calibrate_855042_18.stderr parsl.dataflow.dflow INFO: Task 523 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 591e05ef05082a4ce17e056b437c2d32 with result from task 523 parsl.dataflow.dflow DEBUG: Task 523 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 524 has memoization hash f3b4369f3e85bfa2217eb07e2bba4464 parsl.dataflow.memoization INFO: Task 524 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa457009a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 524 try 0 launched on executor multi with executor id 1013 parsl.dataflow.dflow INFO: Standard output for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/7b970726-4aa7-4347-865d-960b4d7a43f1_calibrate_855137_39.stdout parsl.dataflow.dflow INFO: Standard error for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/7b970726-4aa7-4347-865d-960b4d7a43f1_calibrate_855137_39.stderr parsl.dataflow.dflow INFO: Standard output for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/dd92eb2a-0ec0-4c4c-8b1d-726be4addc7d_characterizeImage_855137_39.stdout parsl.dataflow.dflow INFO: Standard error for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/dd92eb2a-0ec0-4c4c-8b1d-726be4addc7d_characterizeImage_855137_39.stderr parsl.dataflow.dflow INFO: Task 2 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73df80ce85c0976e074cb31fbb1e36a0 with result from task 2 parsl.dataflow.dflow DEBUG: Task 2 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/5f60e8ad-81ab-4855-a2e3-8d87ff625f36_calibrate_855137_15.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/5f60e8ad-81ab-4855-a2e3-8d87ff625f36_calibrate_855137_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:41 proc_lsst.multi:146 [INFO] found job 16912378 in provider checkpoint proc_lsst.multi INFO: found job 16912378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 553 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c48847435dd16c7d3961d4f2c2654ff with result from task 553 parsl.dataflow.dflow DEBUG: Task 553 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 554 has memoization hash fd26b0a701d7fe6bc00a542e042a1f40 parsl.dataflow.memoization INFO: Task 554 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45701e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 554 try 0 launched on executor multi with executor id 1014 parsl.dataflow.dflow INFO: Standard output for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/a013e6a5-fee0-4622-8014-eeaf767c88cb_calibrate_855044_14.stdout parsl.dataflow.dflow INFO: Standard error for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/a013e6a5-fee0-4622-8014-eeaf767c88cb_calibrate_855044_14.stderr parsl.dataflow.dflow INFO: Standard output for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/71f72655-9033-4b00-8b86-2df3538044fe_characterizeImage_855044_14.stdout parsl.dataflow.dflow INFO: Standard error for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/71f72655-9033-4b00-8b86-2df3538044fe_characterizeImage_855044_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 233 parsl.dataflow.strategy DEBUG: Executor multi has 233 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 2024-03-08 07:19:41 proc_lsst.multi:146 [INFO] found job 75775 in provider local proc_lsst.multi INFO: found job 75775 in provider local parsl.dataflow.dflow INFO: Task 476 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 466a67c30f95792bbaed5aaa4a5b8475 with result from task 476 parsl.dataflow.dflow DEBUG: Task 476 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/d7b8bd24-d2bc-41f1-9836-09c59aa44fb1_calibrate_855043_14.stdout parsl.dataflow.dflow INFO: Standard error for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/d7b8bd24-d2bc-41f1-9836-09c59aa44fb1_calibrate_855043_14.stderr parsl.dataflow.dflow INFO: Task 878 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e948897629e422307511c918b926eea3 with result from task 878 parsl.dataflow.dflow DEBUG: Task 878 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/89412d7e-1980-42ce-903d-24097dfcb2ed_calibrate_855137_40.stdout parsl.dataflow.dflow INFO: Standard error for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/89412d7e-1980-42ce-903d-24097dfcb2ed_calibrate_855137_40.stderr 2024-03-08 07:19:41 proc_lsst.multi:146 [INFO] found job 16912283 in provider astro proc_lsst.multi INFO: found job 16912283 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 994 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 335d10d1d33a3afde4dab0c9ffe2a3d5 with result from task 994 parsl.dataflow.dflow DEBUG: Task 994 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 995 has memoization hash 6456f1cdafdffa46e5277fe928bd7f86 parsl.dataflow.memoization INFO: Task 995 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454fb560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 995 try 0 launched on executor multi with executor id 1015 parsl.dataflow.dflow INFO: Standard output for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/039074ff-43f8-4f38-a199-0ed3c4a6719e_calibrate_855136_10.stdout parsl.dataflow.dflow INFO: Standard error for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/039074ff-43f8-4f38-a199-0ed3c4a6719e_calibrate_855136_10.stderr parsl.dataflow.dflow INFO: Standard output for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/85d0f654-ffcc-478a-888b-210b8e3f2258_characterizeImage_855136_10.stdout parsl.dataflow.dflow INFO: Standard error for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/85d0f654-ffcc-478a-888b-210b8e3f2258_characterizeImage_855136_10.stderr parsl.dataflow.dflow INFO: Task 854 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a3935e2b2004fd0e6fea0f6961bd22c with result from task 854 parsl.dataflow.dflow DEBUG: Task 854 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/49dbf0c8-9dbe-4405-85a1-969c39779016_calibrate_855137_16.stdout parsl.dataflow.dflow INFO: Standard error for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/49dbf0c8-9dbe-4405-85a1-969c39779016_calibrate_855137_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:41 proc_lsst.multi:146 [INFO] found job 16912284 in provider astro proc_lsst.multi INFO: found job 16912284 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 905 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9ac90ff4cade1779b5c802c71192dc0 with result from task 905 parsl.dataflow.dflow DEBUG: Task 905 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/db5a35f7-47d6-404e-9c5a-39f321faf180_calibrate_855043_56.stdout parsl.dataflow.dflow INFO: Standard error for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/db5a35f7-47d6-404e-9c5a-39f321faf180_calibrate_855043_56.stderr parsl.dataflow.dflow INFO: Task 808 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b2b7351cb68290536b10ea558a5baa3 with result from task 808 parsl.dataflow.dflow DEBUG: Task 808 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 809 has memoization hash 94bb5e638c9ec99ba060d20f28b6bebb parsl.dataflow.memoization INFO: Task 809 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 809 try 0 launched on executor multi with executor id 1016 parsl.dataflow.dflow INFO: Standard output for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/e9f6cdba-499b-4e69-8076-ac144e700500_calibrate_855137_58.stdout parsl.dataflow.dflow INFO: Standard error for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/e9f6cdba-499b-4e69-8076-ac144e700500_calibrate_855137_58.stderr parsl.dataflow.dflow INFO: Standard output for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/da405938-444c-43fb-86d0-104588ea6697_characterizeImage_855137_58.stdout parsl.dataflow.dflow INFO: Standard error for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/da405938-444c-43fb-86d0-104588ea6697_characterizeImage_855137_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:42 proc_lsst.multi:146 [INFO] found job 16912285 in provider astro proc_lsst.multi INFO: found job 16912285 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 727 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aeaf0a94fc91c8f27d216425ddb33096 with result from task 727 parsl.dataflow.dflow DEBUG: Task 727 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 728 has memoization hash c3ee76155564cc3e803d4889fad9f1b6 parsl.dataflow.memoization INFO: Task 728 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d1c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 728 try 0 launched on executor multi with executor id 1017 parsl.dataflow.dflow INFO: Standard output for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/54696afa-5183-4166-9019-dc0856a68b5b_calibrate_855042_41.stdout parsl.dataflow.dflow INFO: Standard error for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/54696afa-5183-4166-9019-dc0856a68b5b_calibrate_855042_41.stderr parsl.dataflow.dflow INFO: Standard output for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/05c92e93-6692-410c-808c-7e274b772fb3_characterizeImage_855042_41.stdout parsl.dataflow.dflow INFO: Standard error for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/05c92e93-6692-410c-808c-7e274b772fb3_characterizeImage_855042_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:42 proc_lsst.multi:146 [INFO] found job 16912286 in provider astro proc_lsst.multi INFO: found job 16912286 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:42 proc_lsst.multi:146 [INFO] found job 16912287 in provider astro proc_lsst.multi INFO: found job 16912287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 611 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e09da0234e4e4e06b55a92af458fc255 with result from task 611 parsl.dataflow.dflow DEBUG: Task 611 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/1203235c-f5ae-4a01-8b18-76caa38678c1_calibrate_855044_46.stdout parsl.dataflow.dflow INFO: Standard error for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/1203235c-f5ae-4a01-8b18-76caa38678c1_calibrate_855044_46.stderr parsl.dataflow.dflow INFO: Task 973 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb0974975f8132aa031b8640487a1f85 with result from task 973 parsl.dataflow.dflow DEBUG: Task 973 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 974 has memoization hash 778dbb2c18dea7a783ab1b44ee16419b parsl.dataflow.memoization INFO: Task 974 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454f8d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 974 try 0 launched on executor multi with executor id 1018 parsl.dataflow.dflow INFO: Standard output for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/66e5cea6-1084-43f6-8774-32b68fd5d24d_calibrate_855137_10.stdout parsl.dataflow.dflow INFO: Standard error for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/66e5cea6-1084-43f6-8774-32b68fd5d24d_calibrate_855137_10.stderr parsl.dataflow.dflow INFO: Standard output for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/75e3df4b-def6-457e-84bb-d962df526faa_characterizeImage_855137_10.stdout parsl.dataflow.dflow INFO: Standard error for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/75e3df4b-def6-457e-84bb-d962df526faa_characterizeImage_855137_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:43 proc_lsst.multi:146 [INFO] found job 16912288 in provider astro proc_lsst.multi INFO: found job 16912288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 917 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 551beeb44f7de16d6a90a6a28f364758 with result from task 917 parsl.dataflow.dflow DEBUG: Task 917 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/d8ca358d-c1dc-46c6-8765-28e394f0a26a_calibrate_855136_57.stdout parsl.dataflow.dflow INFO: Standard error for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/d8ca358d-c1dc-46c6-8765-28e394f0a26a_calibrate_855136_57.stderr parsl.dataflow.dflow INFO: Task 491 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 88b60c57d1345c554214e7b52493d818 with result from task 491 parsl.dataflow.dflow DEBUG: Task 491 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/889a4534-172b-4a14-9126-f4db664621dc_calibrate_855044_25.stdout parsl.dataflow.dflow INFO: Standard error for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/889a4534-172b-4a14-9126-f4db664621dc_calibrate_855044_25.stderr parsl.dataflow.dflow INFO: Task 767 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cff9678e815c43d1cf9fb3881705b5e6 with result from task 767 parsl.dataflow.dflow DEBUG: Task 767 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/1fcea252-194e-4bfb-a3f4-ed61ae09420a_calibrate_855137_44.stdout parsl.dataflow.dflow INFO: Standard error for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/1fcea252-194e-4bfb-a3f4-ed61ae09420a_calibrate_855137_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:43 proc_lsst.multi:146 [INFO] found job 16912289 in provider astro proc_lsst.multi INFO: found job 16912289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 968 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce1eaf46384e0c1f0698f5c17850297a with result from task 968 parsl.dataflow.dflow DEBUG: Task 968 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/5a96b929-1001-47a8-91f6-835a2723ac82_calibrate_855043_7.stdout parsl.dataflow.dflow INFO: Standard error for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/5a96b929-1001-47a8-91f6-835a2723ac82_calibrate_855043_7.stderr parsl.dataflow.dflow INFO: Task 59 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9dfcb4a7f31ce1df2970a33cc38cb7bf with result from task 59 parsl.dataflow.dflow DEBUG: Task 59 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/582afeb0-3809-4d7d-bdba-cb1481c6b075_calibrate_855136_62.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/582afeb0-3809-4d7d-bdba-cb1481c6b075_calibrate_855136_62.stderr parsl.dataflow.dflow INFO: Task 893 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce30f26b8da66565d8b9a962c873b28f with result from task 893 parsl.dataflow.dflow DEBUG: Task 893 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/6f582040-766c-4073-811a-a423f44e64f7_calibrate_855138_40.stdout parsl.dataflow.dflow INFO: Standard error for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/6f582040-766c-4073-811a-a423f44e64f7_calibrate_855138_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:43 proc_lsst.multi:146 [INFO] found job 16912290 in provider astro proc_lsst.multi INFO: found job 16912290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 77 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b32b711c445421797c05af8f7f2951d with result from task 77 parsl.dataflow.dflow DEBUG: Task 77 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/8a3a873e-13dc-4b01-a857-42d0ce6a273d_calibrate_855136_44.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/8a3a873e-13dc-4b01-a857-42d0ce6a273d_calibrate_855136_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:44 proc_lsst.multi:146 [INFO] found job 16912291 in provider astro proc_lsst.multi INFO: found job 16912291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 529 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8552d79791080e65f8b20fa93c620c12 with result from task 529 parsl.dataflow.dflow DEBUG: Task 529 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 530 has memoization hash 3ecefbf10bb97baede711c771626c44b parsl.dataflow.memoization INFO: Task 530 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45703ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 530 try 0 launched on executor multi with executor id 1019 parsl.dataflow.dflow INFO: Standard output for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/2d3dd20f-1c88-4d5f-924e-94cffcb81dd3_calibrate_855044_38.stdout parsl.dataflow.dflow INFO: Standard error for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/2d3dd20f-1c88-4d5f-924e-94cffcb81dd3_calibrate_855044_38.stderr parsl.dataflow.dflow INFO: Standard output for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/389e2677-d604-47c9-976f-31cb1ab0bb10_characterizeImage_855044_38.stdout parsl.dataflow.dflow INFO: Standard error for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/389e2677-d604-47c9-976f-31cb1ab0bb10_characterizeImage_855044_38.stderr parsl.dataflow.dflow INFO: Task 5 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5d41956feb51c0235da461af9ec39d0 with result from task 5 parsl.dataflow.dflow DEBUG: Task 5 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/1e2af3ee-d90c-4e00-b5be-54f7461249b0_calibrate_855137_13.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/1e2af3ee-d90c-4e00-b5be-54f7461249b0_calibrate_855137_13.stderr parsl.dataflow.dflow INFO: Task 517 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ff0a66e63eba8c89d77d92f9febc568 with result from task 517 parsl.dataflow.dflow DEBUG: Task 517 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 518 has memoization hash 11fd04f6d9590a27854dde9d43a57c3b parsl.dataflow.memoization INFO: Task 518 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa457002c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 518 try 0 launched on executor multi with executor id 1020 parsl.dataflow.dflow INFO: Standard output for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/f45890c3-8a7b-4f26-89f9-cb3255418d02_calibrate_855137_17.stdout parsl.dataflow.dflow INFO: Standard error for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/f45890c3-8a7b-4f26-89f9-cb3255418d02_calibrate_855137_17.stderr parsl.dataflow.dflow INFO: Standard output for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/1da18e3e-dea3-4a53-8e8d-8334bc0d616e_characterizeImage_855137_17.stdout parsl.dataflow.dflow INFO: Standard error for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/1da18e3e-dea3-4a53-8e8d-8334bc0d616e_characterizeImage_855137_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:44 proc_lsst.multi:146 [INFO] found job 16912292 in provider astro proc_lsst.multi INFO: found job 16912292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 520 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4f21d109f3cbf6049589a3efe06b866 with result from task 520 parsl.dataflow.dflow DEBUG: Task 520 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 521 has memoization hash 70f41e38648e407d190a7931f5f74ed6 parsl.dataflow.memoization INFO: Task 521 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45700220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 521 try 0 launched on executor multi with executor id 1021 parsl.dataflow.dflow INFO: Standard output for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/47d8eb36-40b9-40df-bf82-dea76991dfd4_calibrate_855137_41.stdout parsl.dataflow.dflow INFO: Standard error for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/47d8eb36-40b9-40df-bf82-dea76991dfd4_calibrate_855137_41.stderr parsl.dataflow.dflow INFO: Standard output for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/cb241c15-812f-4969-b025-c98ca1e7123d_characterizeImage_855137_41.stdout parsl.dataflow.dflow INFO: Standard error for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/cb241c15-812f-4969-b025-c98ca1e7123d_characterizeImage_855137_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:44 proc_lsst.multi:146 [INFO] found job 16912293 in provider astro proc_lsst.multi INFO: found job 16912293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 212 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72c890fdf395158042e984d6c65eae5c with result from task 212 parsl.dataflow.dflow DEBUG: Task 212 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/f6a639d1-d57c-4194-af04-1c5749b8dafa_calibrate_855137_33.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/f6a639d1-d57c-4194-af04-1c5749b8dafa_calibrate_855137_33.stderr parsl.dataflow.dflow INFO: Task 617 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63752b44d97e2d4dce829678c93949e4 with result from task 617 parsl.dataflow.dflow DEBUG: Task 617 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/944f0ef1-c6da-4213-b19b-d10b7af6e7ee_calibrate_855136_49.stdout parsl.dataflow.dflow INFO: Standard error for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/944f0ef1-c6da-4213-b19b-d10b7af6e7ee_calibrate_855136_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:45 proc_lsst.multi:146 [INFO] found job 16912294 in provider astro proc_lsst.multi INFO: found job 16912294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 653 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e981ec66bd824c44c6eb6bb559d79f0 with result from task 653 parsl.dataflow.dflow DEBUG: Task 653 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/e244a8b4-d1e9-4c2a-8a4f-376fd4861b95_calibrate_855042_9.stdout parsl.dataflow.dflow INFO: Standard error for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/e244a8b4-d1e9-4c2a-8a4f-376fd4861b95_calibrate_855042_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:45 proc_lsst.multi:146 [INFO] found job 16912295 in provider astro proc_lsst.multi INFO: found job 16912295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:45 proc_lsst.multi:146 [INFO] found job 16912296 in provider astro proc_lsst.multi INFO: found job 16912296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:46 proc_lsst.multi:146 [INFO] found job 16912297 in provider astro proc_lsst.multi INFO: found job 16912297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 752 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry adcf5eb84dea58b461e4f535eb28a1d1 with result from task 752 parsl.dataflow.dflow DEBUG: Task 752 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/971aff00-e271-4961-8bdd-2057d995c57b_calibrate_855138_44.stdout parsl.dataflow.dflow INFO: Standard error for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/971aff00-e271-4961-8bdd-2057d995c57b_calibrate_855138_44.stderr parsl.dataflow.dflow INFO: Task 890 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed35f1563f192bad6e19c9cea28fa6d8 with result from task 890 parsl.dataflow.dflow DEBUG: Task 890 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/2473ae04-b3fe-4f3d-9f3d-5628af1c1084_calibrate_855043_52.stdout parsl.dataflow.dflow INFO: Standard error for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/2473ae04-b3fe-4f3d-9f3d-5628af1c1084_calibrate_855043_52.stderr parsl.dataflow.dflow INFO: Task 647 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 007228ffd4136869e6ffd0906871ebbe with result from task 647 parsl.dataflow.dflow DEBUG: Task 647 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/5f59fbdb-5863-4074-826c-779094b522f8_calibrate_855043_44.stdout parsl.dataflow.dflow INFO: Standard error for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/5f59fbdb-5863-4074-826c-779094b522f8_calibrate_855043_44.stderr parsl.dataflow.dflow INFO: Task 593 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 357f490e236c70c55819cf542b319e22 with result from task 593 parsl.dataflow.dflow DEBUG: Task 593 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/3ca973ff-378e-44bd-bc49-f3798086fb4e_calibrate_855138_43.stdout parsl.dataflow.dflow INFO: Standard error for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/3ca973ff-378e-44bd-bc49-f3798086fb4e_calibrate_855138_43.stderr parsl.dataflow.dflow INFO: Task 575 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f34e4f4b6db1ebf96801c1f1aba765e3 with result from task 575 parsl.dataflow.dflow DEBUG: Task 575 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/580c67e8-88f5-4073-a324-52ad94f043f5_calibrate_855138_6.stdout parsl.dataflow.dflow INFO: Standard error for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/580c67e8-88f5-4073-a324-52ad94f043f5_calibrate_855138_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:46 proc_lsst.multi:146 [INFO] found job 16912298 in provider astro proc_lsst.multi INFO: found job 16912298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 307 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c9fc7a70df39000b497dcce1784378b with result from task 307 parsl.dataflow.dflow DEBUG: Task 307 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 308 has memoization hash f73a2e7a86aeb111dca217b2f9390228 parsl.dataflow.memoization INFO: Task 308 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 308 try 0 launched on executor multi with executor id 1022 parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/65b4adf5-c1fe-4bfa-9f03-96ae43cdb0de_calibrate_855042_4.stdout parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/65b4adf5-c1fe-4bfa-9f03-96ae43cdb0de_calibrate_855042_4.stderr parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/288f0547-cb42-4a8b-804b-071ef0cc886f_characterizeImage_855042_4.stdout parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/288f0547-cb42-4a8b-804b-071ef0cc886f_characterizeImage_855042_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:46 proc_lsst.multi:146 [INFO] found job 16912299 in provider astro proc_lsst.multi INFO: found job 16912299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 311 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6270507f00689182ea2f9bf1264ddf77 with result from task 311 parsl.dataflow.dflow DEBUG: Task 311 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/9dd0836d-1636-4357-a157-b9c4630a86f4_calibrate_855043_42.stdout parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/9dd0836d-1636-4357-a157-b9c4630a86f4_calibrate_855043_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:47 proc_lsst.multi:146 [INFO] found job 16912300 in provider astro proc_lsst.multi INFO: found job 16912300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:47 proc_lsst.multi:146 [INFO] found job 16912301 in provider astro proc_lsst.multi INFO: found job 16912301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 185 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 43e179cb8f657e8cee64d8ddf3202595 with result from task 185 parsl.dataflow.dflow DEBUG: Task 185 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/b3dab249-30f0-472d-859b-9f30821f05db_calibrate_855137_6.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/b3dab249-30f0-472d-859b-9f30821f05db_calibrate_855137_6.stderr parsl.dataflow.dflow INFO: Task 1114 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 737803ac1b243421bbcc8f5466e7d364 with result from task 1114 parsl.dataflow.dflow DEBUG: Task 1114 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1115 has memoization hash 82323accc9a3f65e1979679da3356d1b parsl.dataflow.memoization INFO: Task 1115 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45391f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1115 try 0 launched on executor multi with executor id 1023 parsl.dataflow.dflow INFO: Standard output for task 1115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/4b4c4f5e-4550-475f-a4fa-813de3fc489e_calibrate_855137_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/4b4c4f5e-4550-475f-a4fa-813de3fc489e_calibrate_855137_56.stderr parsl.dataflow.dflow INFO: Standard output for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/968e1097-1bbd-49c4-98d9-71ca561a4353_characterizeImage_855137_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/968e1097-1bbd-49c4-98d9-71ca561a4353_characterizeImage_855137_56.stderr parsl.dataflow.dflow INFO: Task 932 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry caa4443c2e938dbfbf700f10a8cfb86b with result from task 932 parsl.dataflow.dflow DEBUG: Task 932 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/1b6b1ab7-f2c5-4ded-b5ce-ba89f56e4cf9_calibrate_855136_55.stdout parsl.dataflow.dflow INFO: Standard error for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/1b6b1ab7-f2c5-4ded-b5ce-ba89f56e4cf9_calibrate_855136_55.stderr parsl.dataflow.dflow INFO: Task 248 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f5edf0cd4c037d759d72fd476149f71 with result from task 248 parsl.dataflow.dflow DEBUG: Task 248 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/8883a743-dbbb-4d6b-bf1a-a632f3a5854a_calibrate_855136_48.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/8883a743-dbbb-4d6b-bf1a-a632f3a5854a_calibrate_855136_48.stderr parsl.dataflow.dflow INFO: Task 487 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7db1acb168c73964a19cd57ae028411e with result from task 487 parsl.dataflow.dflow DEBUG: Task 487 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 488 has memoization hash 93471f5f5ff8b4b77d444511abbb010e parsl.dataflow.memoization INFO: Task 488 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456cd3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 488 try 0 launched on executor multi with executor id 1024 parsl.dataflow.dflow INFO: Standard output for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/e3dc2cd0-6814-4a57-90b0-f0a4b0f8e82d_calibrate_855137_35.stdout parsl.dataflow.dflow INFO: Standard error for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/e3dc2cd0-6814-4a57-90b0-f0a4b0f8e82d_calibrate_855137_35.stderr parsl.dataflow.dflow INFO: Standard output for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/3257157e-c0c8-4704-883b-4a527a9f89e1_characterizeImage_855137_35.stdout parsl.dataflow.dflow INFO: Standard error for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/3257157e-c0c8-4704-883b-4a527a9f89e1_characterizeImage_855137_35.stderr parsl.dataflow.dflow INFO: Task 692 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8deb930f1b053404d421a7762ae554b4 with result from task 692 parsl.dataflow.dflow DEBUG: Task 692 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/173f3aae-4d0e-482d-ab51-4f779788cf61_calibrate_855136_20.stdout parsl.dataflow.dflow INFO: Standard error for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/173f3aae-4d0e-482d-ab51-4f779788cf61_calibrate_855136_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:47 proc_lsst.multi:146 [INFO] found job 16912302 in provider astro proc_lsst.multi INFO: found job 16912302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 359 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f0ae943ceddb23ad673216464ee80c58 with result from task 359 parsl.dataflow.dflow DEBUG: Task 359 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/0e80ede9-c775-4be3-b2fc-bb9609365bd4_calibrate_855042_34.stdout parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/0e80ede9-c775-4be3-b2fc-bb9609365bd4_calibrate_855042_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:48 proc_lsst.multi:146 [INFO] found job 16912303 in provider astro proc_lsst.multi INFO: found job 16912303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 35 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1605addffc9d50970752dbdf7f6a3cb6 with result from task 35 parsl.dataflow.dflow DEBUG: Task 35 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/1028dae4-a45e-447e-800b-e0a7f158a0e1_calibrate_855137_7.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/1028dae4-a45e-447e-800b-e0a7f158a0e1_calibrate_855137_7.stderr parsl.dataflow.dflow INFO: Task 62 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e71bc1cb7ed71638e048ff99085fc216 with result from task 62 parsl.dataflow.dflow DEBUG: Task 62 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/9e5189fa-751c-47f5-a214-52eaa99e8faa_calibrate_855138_60.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/9e5189fa-751c-47f5-a214-52eaa99e8faa_calibrate_855138_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:48 proc_lsst.multi:146 [INFO] found job 16912304 in provider astro proc_lsst.multi INFO: found job 16912304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 1102 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8fba6ac30cfd3dc99f53fce6ccd2ccf with result from task 1102 parsl.dataflow.dflow DEBUG: Task 1102 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1103 has memoization hash a47667438e8e4b8acc857599bda52926 parsl.dataflow.memoization INFO: Task 1103 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45391b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1103 try 0 launched on executor multi with executor id 1025 parsl.dataflow.dflow INFO: Standard output for task 1103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/c1743a72-877a-4b81-a62c-5d9cd144c3ee_calibrate_855136_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/c1743a72-877a-4b81-a62c-5d9cd144c3ee_calibrate_855136_18.stderr parsl.dataflow.dflow INFO: Standard output for task 1102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/95e4f8e6-3262-44dd-9945-ca0163e2da91_characterizeImage_855136_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/95e4f8e6-3262-44dd-9945-ca0163e2da91_characterizeImage_855136_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:48 proc_lsst.multi:146 [INFO] found job 16912305 in provider astro proc_lsst.multi INFO: found job 16912305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:49 proc_lsst.multi:146 [INFO] found job 16912306 in provider astro proc_lsst.multi INFO: found job 16912306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:49 proc_lsst.multi:146 [INFO] found job 16912307 in provider astro proc_lsst.multi INFO: found job 16912307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 667 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98ea1bca8cd3e25f982427233db3d17d with result from task 667 parsl.dataflow.dflow DEBUG: Task 667 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 668 has memoization hash 9185b08a02566eca5ff96cb520b54e57 parsl.dataflow.memoization INFO: Task 668 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 668 try 0 launched on executor multi with executor id 1026 parsl.dataflow.dflow INFO: Standard output for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/e14b885e-59a4-4e1d-bf95-b0a929761b58_calibrate_855043_18.stdout parsl.dataflow.dflow INFO: Standard error for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/e14b885e-59a4-4e1d-bf95-b0a929761b58_calibrate_855043_18.stderr parsl.dataflow.dflow INFO: Standard output for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/e1119258-8d35-4ea0-8f58-2f4924344592_characterizeImage_855043_18.stdout parsl.dataflow.dflow INFO: Standard error for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/e1119258-8d35-4ea0-8f58-2f4924344592_characterizeImage_855043_18.stderr parsl.dataflow.dflow INFO: Task 698 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4fd3d52870cf687cc42143240affbd14 with result from task 698 parsl.dataflow.dflow DEBUG: Task 698 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/52121c8e-ac69-4456-b5be-d5b07bb057e7_calibrate_855137_8.stdout parsl.dataflow.dflow INFO: Standard error for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/52121c8e-ac69-4456-b5be-d5b07bb057e7_calibrate_855137_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:49 proc_lsst.multi:146 [INFO] found job 16912308 in provider astro proc_lsst.multi INFO: found job 16912308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:50 proc_lsst.multi:146 [INFO] found job 16912309 in provider astro proc_lsst.multi INFO: found job 16912309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 257 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2e098e2bb529a62016b58b68bc38b17 with result from task 257 parsl.dataflow.dflow DEBUG: Task 257 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/a6621236-1fa4-4a66-bd98-7c70f48c36f7_calibrate_855042_43.stdout parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/a6621236-1fa4-4a66-bd98-7c70f48c36f7_calibrate_855042_43.stderr parsl.dataflow.dflow INFO: Task 1030 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33a4f2ab1c376f491a52a029327765e2 with result from task 1030 parsl.dataflow.dflow DEBUG: Task 1030 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1031 has memoization hash fb0de815c72835b00261693303d57a1e parsl.dataflow.memoization INFO: Task 1031 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45541760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1031 try 0 launched on executor multi with executor id 1027 parsl.dataflow.dflow INFO: Standard output for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/7d9f11ce-4800-43e3-8a87-5ae971b1b8b5_calibrate_855042_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/7d9f11ce-4800-43e3-8a87-5ae971b1b8b5_calibrate_855042_40.stderr parsl.dataflow.dflow INFO: Standard output for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/19ebfe5c-9f78-4413-a9b8-f111de721222_characterizeImage_855042_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/19ebfe5c-9f78-4413-a9b8-f111de721222_characterizeImage_855042_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:50 proc_lsst.multi:146 [INFO] found job 16912310 in provider astro proc_lsst.multi INFO: found job 16912310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:50 proc_lsst.multi:146 [INFO] found job 16912311 in provider astro proc_lsst.multi INFO: found job 16912311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 179 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d2a7999a1a0e97490b94f4511e24bbfc with result from task 179 parsl.dataflow.dflow DEBUG: Task 179 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/1074de16-0c6b-4d47-ade9-2c9c951b36da_calibrate_855137_62.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/1074de16-0c6b-4d47-ade9-2c9c951b36da_calibrate_855137_62.stderr parsl.dataflow.dflow INFO: Task 509 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf5175b53a6565d4408858f75da61bb3 with result from task 509 parsl.dataflow.dflow DEBUG: Task 509 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/115b8f32-d876-48e5-9ff6-6a05bdd7f56a_calibrate_855044_27.stdout parsl.dataflow.dflow INFO: Standard error for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/115b8f32-d876-48e5-9ff6-6a05bdd7f56a_calibrate_855044_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:51 proc_lsst.multi:146 [INFO] found job 16912312 in provider astro proc_lsst.multi INFO: found job 16912312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 535 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf4605cef489954be5580490b2014b59 with result from task 535 parsl.dataflow.dflow DEBUG: Task 535 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 536 has memoization hash 6ef0b7589940c9c848411fc55eb7ec4c parsl.dataflow.memoization INFO: Task 536 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa457037e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 536 try 0 launched on executor multi with executor id 1028 parsl.dataflow.dflow INFO: Standard output for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/6103caea-c873-4c47-9ab4-5164de9ec718_calibrate_855042_7.stdout parsl.dataflow.dflow INFO: Standard error for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/6103caea-c873-4c47-9ab4-5164de9ec718_calibrate_855042_7.stderr parsl.dataflow.dflow INFO: Standard output for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/6669a619-ccd6-4d0b-a6be-d89d794f9d2d_characterizeImage_855042_7.stdout parsl.dataflow.dflow INFO: Standard error for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/6669a619-ccd6-4d0b-a6be-d89d794f9d2d_characterizeImage_855042_7.stderr parsl.dataflow.dflow INFO: Task 329 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 45a674d885c5ce9c752893da181ec75a with result from task 329 parsl.dataflow.dflow DEBUG: Task 329 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/839f676e-715a-4d53-989c-57288b0ecede_calibrate_855138_58.stdout parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/839f676e-715a-4d53-989c-57288b0ecede_calibrate_855138_58.stderr parsl.dataflow.dflow INFO: Task 562 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a38c4bb80ffd239860a4eee7fb5ed745 with result from task 562 parsl.dataflow.dflow DEBUG: Task 562 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 563 has memoization hash 325867948d6ce9e523a09dbab0a90262 parsl.dataflow.memoization INFO: Task 563 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa457028e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 563 try 0 launched on executor multi with executor id 1029 parsl.dataflow.dflow INFO: Standard output for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/270f9954-01b9-465b-bfbd-3ad437337cf1_calibrate_855138_16.stdout parsl.dataflow.dflow INFO: Standard error for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/270f9954-01b9-465b-bfbd-3ad437337cf1_calibrate_855138_16.stderr parsl.dataflow.dflow INFO: Standard output for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/0ed78163-9a35-478f-9e9b-1efab5d52fe7_characterizeImage_855138_16.stdout parsl.dataflow.dflow INFO: Standard error for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/0ed78163-9a35-478f-9e9b-1efab5d52fe7_characterizeImage_855138_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:51 proc_lsst.multi:146 [INFO] found job 16912313 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912313 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.dataflow.dflow INFO: Task 860 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2584a833cb49451154e5ea2dad3ad723 with result from task 860 parsl.dataflow.dflow DEBUG: Task 860 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/bcaf17dc-c1bb-419b-86be-53a5693a78c1_calibrate_855136_32.stdout parsl.dataflow.dflow INFO: Standard error for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/bcaf17dc-c1bb-419b-86be-53a5693a78c1_calibrate_855136_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:51 proc_lsst.multi:146 [INFO] found job 16912314 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912314 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.dataflow.dflow INFO: Task 1003 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50ccfeca0b903a9ed991ae8a6767966f with result from task 1003 parsl.dataflow.dflow DEBUG: Task 1003 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1004 has memoization hash 03673402609d52cf6f8a28a533a5ac04 parsl.dataflow.memoization INFO: Task 1004 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454fbe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1004 try 0 launched on executor multi with executor id 1030 parsl.dataflow.dflow INFO: Standard output for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/f64addfc-6f3e-4ffe-909f-b6a997265498_calibrate_855136_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/f64addfc-6f3e-4ffe-909f-b6a997265498_calibrate_855136_53.stderr parsl.dataflow.dflow INFO: Standard output for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/01b8cd61-948a-4d2c-8453-ebb5f5914407_characterizeImage_855136_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/01b8cd61-948a-4d2c-8453-ebb5f5914407_characterizeImage_855136_53.stderr parsl.dataflow.dflow INFO: Task 730 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cbc233fedaea1ed123de11cbf3048a49 with result from task 730 parsl.dataflow.dflow DEBUG: Task 730 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 731 has memoization hash 112d83c3c0eeca083a17449840837cb2 parsl.dataflow.memoization INFO: Task 731 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d1f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 731 try 0 launched on executor multi with executor id 1031 parsl.dataflow.dflow INFO: Standard output for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/dd9afb08-9554-45ee-87d7-10711e1f4c9e_calibrate_855044_47.stdout parsl.dataflow.dflow INFO: Standard error for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/dd9afb08-9554-45ee-87d7-10711e1f4c9e_calibrate_855044_47.stderr parsl.dataflow.dflow INFO: Standard output for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/12f32eb7-6d29-4620-88e7-b28c0ea271fd_characterizeImage_855044_47.stdout parsl.dataflow.dflow INFO: Standard error for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/12f32eb7-6d29-4620-88e7-b28c0ea271fd_characterizeImage_855044_47.stderr parsl.dataflow.dflow INFO: Task 842 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5455fb0ecef468fdc5d5dd29c344da1 with result from task 842 parsl.dataflow.dflow DEBUG: Task 842 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/19a8c8fb-74ea-4bd8-a7ef-30f637fc0451_calibrate_855137_42.stdout parsl.dataflow.dflow INFO: Standard error for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/19a8c8fb-74ea-4bd8-a7ef-30f637fc0451_calibrate_855137_42.stderr parsl.dataflow.dflow INFO: Task 374 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e25366e944f6df83ddcc3f03fb3e4056 with result from task 374 parsl.dataflow.dflow DEBUG: Task 374 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/094bc063-3328-43d0-afca-b0cc248491b1_calibrate_855043_47.stdout parsl.dataflow.dflow INFO: Standard error for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/094bc063-3328-43d0-afca-b0cc248491b1_calibrate_855043_47.stderr parsl.dataflow.dflow INFO: Task 1009 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c0b00c436ea77c889f4896474f83a5f8 with result from task 1009 parsl.dataflow.dflow DEBUG: Task 1009 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1010 has memoization hash f787a6dbf0872c1b5c6262ea9a372207 parsl.dataflow.memoization INFO: Task 1010 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45540220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1010 try 0 launched on executor multi with executor id 1032 parsl.dataflow.dflow INFO: Standard output for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/0b9289e1-11ef-498d-bf73-144de8c1c8a5_calibrate_855043_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/0b9289e1-11ef-498d-bf73-144de8c1c8a5_calibrate_855043_24.stderr parsl.dataflow.dflow INFO: Standard output for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/16c1c28d-d8d6-4d5d-a71c-c6089656cef7_characterizeImage_855043_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/16c1c28d-d8d6-4d5d-a71c-c6089656cef7_characterizeImage_855043_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:52 proc_lsst.multi:146 [INFO] found job 16912315 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912315 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.dataflow.dflow INFO: Task 404 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b1921eb8435a473fc09f5b2893d95fa3 with result from task 404 parsl.dataflow.dflow DEBUG: Task 404 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/53d967a1-fa3b-4b0e-b685-8825365da314_calibrate_855136_43.stdout parsl.dataflow.dflow INFO: Standard error for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/53d967a1-fa3b-4b0e-b685-8825365da314_calibrate_855136_43.stderr parsl.dataflow.dflow INFO: Task 44 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7019fbbb186cba8d2fb243d4c90cf829 with result from task 44 parsl.dataflow.dflow DEBUG: Task 44 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/de8e9723-7dcd-49e4-861d-754e99cd4eaf_calibrate_855136_35.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/de8e9723-7dcd-49e4-861d-754e99cd4eaf_calibrate_855136_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:52 proc_lsst.multi:146 [INFO] found job 16912316 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912316 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.dataflow.dflow INFO: Task 368 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 539a109af666bf331021a5b4c658656c with result from task 368 parsl.dataflow.dflow DEBUG: Task 368 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/8a8ce4f8-7d92-4135-85eb-9cd5ebc26ff7_calibrate_855044_17.stdout parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/8a8ce4f8-7d92-4135-85eb-9cd5ebc26ff7_calibrate_855044_17.stderr parsl.dataflow.dflow INFO: Task 466 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef2268178ccb9862c2bc6f168833265b with result from task 466 parsl.dataflow.dflow DEBUG: Task 466 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 467 has memoization hash dd57a732f88240f62a3a7d66f6cfc8d7 parsl.dataflow.memoization INFO: Task 467 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456cf6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 467 try 0 launched on executor multi with executor id 1033 parsl.dataflow.dflow INFO: Standard output for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/4b4b0e2c-cf0c-484f-b029-9370c91e8647_calibrate_855042_30.stdout parsl.dataflow.dflow INFO: Standard error for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/4b4b0e2c-cf0c-484f-b029-9370c91e8647_calibrate_855042_30.stderr parsl.dataflow.dflow INFO: Standard output for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/181b800d-7e08-4612-8667-b99f5074ad3d_characterizeImage_855042_30.stdout parsl.dataflow.dflow INFO: Standard error for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/181b800d-7e08-4612-8667-b99f5074ad3d_characterizeImage_855042_30.stderr parsl.dataflow.dflow INFO: Task 236 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9ac7fc5591f25ad4ad9b643cb32cf25 with result from task 236 parsl.dataflow.dflow DEBUG: Task 236 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/4aa15a4c-fd0d-46cd-aef5-c136cfc61e99_calibrate_855044_16.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/4aa15a4c-fd0d-46cd-aef5-c136cfc61e99_calibrate_855044_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:52 proc_lsst.multi:146 [INFO] found job 16912317 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.dataflow.dflow INFO: Task 86 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f321e011ef317e49bca83236e5d4a1b7 with result from task 86 parsl.dataflow.dflow DEBUG: Task 86 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/4a67262a-7c79-4836-8acf-d390648b703a_calibrate_855137_21.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/4a67262a-7c79-4836-8acf-d390648b703a_calibrate_855137_21.stderr parsl.dataflow.dflow INFO: Task 17 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7dcd3cb6421e4a3506663f59b0ba5d1c with result from task 17 parsl.dataflow.dflow DEBUG: Task 17 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/c9157d28-0381-4d0c-a20c-2d8b608945fb_calibrate_855044_53.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/c9157d28-0381-4d0c-a20c-2d8b608945fb_calibrate_855044_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:53 proc_lsst.multi:146 [INFO] found job 16912318 in provider escience_a40 proc_lsst.multi INFO: found job 16912318 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:53 proc_lsst.multi:146 [INFO] found job 16912319 in provider escience_a40 proc_lsst.multi INFO: found job 16912319 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 437 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f97fc699ebf3eba0a8bb7712607f6f9f with result from task 437 parsl.dataflow.dflow DEBUG: Task 437 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/491b4b4d-32e3-4aa1-baa3-e1ac3d73b02f_calibrate_855137_14.stdout parsl.dataflow.dflow INFO: Standard error for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/491b4b4d-32e3-4aa1-baa3-e1ac3d73b02f_calibrate_855137_14.stderr parsl.dataflow.dflow INFO: Task 839 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bfaedf7286382bffb3269ea398538b4a with result from task 839 parsl.dataflow.dflow DEBUG: Task 839 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/d68242b7-d4a7-4b4a-9e67-2347927c5344_calibrate_855136_9.stdout parsl.dataflow.dflow INFO: Standard error for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/d68242b7-d4a7-4b4a-9e67-2347927c5344_calibrate_855136_9.stderr parsl.dataflow.dflow INFO: Task 1054 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e638114a4a6f25400844d23175a5d083 with result from task 1054 parsl.dataflow.dflow DEBUG: Task 1054 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1055 has memoization hash bbb172b2813927a6a5e26b675c58032c parsl.dataflow.memoization INFO: Task 1055 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45542de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1055 try 0 launched on executor multi with executor id 1034 parsl.dataflow.dflow INFO: Standard output for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/7f9c6961-37fe-4a82-8095-1a064d978b8a_calibrate_855043_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/7f9c6961-37fe-4a82-8095-1a064d978b8a_calibrate_855043_9.stderr parsl.dataflow.dflow INFO: Standard output for task 1054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/b9894a9c-e2bc-45cd-91d0-367729e4463f_characterizeImage_855043_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/b9894a9c-e2bc-45cd-91d0-367729e4463f_characterizeImage_855043_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:53 proc_lsst.multi:146 [INFO] found job 16912320 in provider escience_a40 proc_lsst.multi INFO: found job 16912320 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 23 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b5ab7932944601e606ab36feca69e32 with result from task 23 parsl.dataflow.dflow DEBUG: Task 23 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/2f40fd4f-1aaa-48a8-adf4-6c1c48f251b0_calibrate_855137_52.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/2f40fd4f-1aaa-48a8-adf4-6c1c48f251b0_calibrate_855137_52.stderr parsl.dataflow.dflow INFO: Task 965 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b7f9eed2bac533fcaf88616d40655601 with result from task 965 parsl.dataflow.dflow DEBUG: Task 965 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/237cfd05-6de1-446d-9ef7-fa521434be1c_calibrate_855043_54.stdout parsl.dataflow.dflow INFO: Standard error for task 965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/237cfd05-6de1-446d-9ef7-fa521434be1c_calibrate_855043_54.stderr parsl.dataflow.dflow INFO: Task 464 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3bdfa8f1440ff8d494875bf4b453c7d8 with result from task 464 parsl.dataflow.dflow DEBUG: Task 464 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/015bb548-2c65-4c6a-b486-99522e1e4747_calibrate_855136_40.stdout parsl.dataflow.dflow INFO: Standard error for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/015bb548-2c65-4c6a-b486-99522e1e4747_calibrate_855136_40.stderr parsl.dataflow.dflow INFO: Task 128 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fba1ce37e6808ada60f92ad403a2ee46 with result from task 128 parsl.dataflow.dflow DEBUG: Task 128 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/fc13d0dc-b5b1-4fde-a3f6-684626c2e7aa_calibrate_855136_42.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/fc13d0dc-b5b1-4fde-a3f6-684626c2e7aa_calibrate_855136_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:53 proc_lsst.multi:146 [INFO] found job 16912321 in provider escience_a40 proc_lsst.multi INFO: found job 16912321 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 278 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f1804e83c06a2805b48057f40680184 with result from task 278 parsl.dataflow.dflow DEBUG: Task 278 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/4a32354a-fef0-4d69-9dd9-73854af53438_calibrate_855137_36.stdout parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/4a32354a-fef0-4d69-9dd9-73854af53438_calibrate_855137_36.stderr parsl.dataflow.dflow INFO: Task 74 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3aadf7902866e0c66578685c90c88bd with result from task 74 parsl.dataflow.dflow DEBUG: Task 74 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/6f1e82dd-ea9c-4e17-8ca0-fc8d408f2e45_calibrate_855136_13.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/6f1e82dd-ea9c-4e17-8ca0-fc8d408f2e45_calibrate_855136_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:54 proc_lsst.multi:146 [INFO] found job 16912322 in provider escience_a40 proc_lsst.multi INFO: found job 16912322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:54 proc_lsst.multi:146 [INFO] found job 16912323 in provider escience_a40 proc_lsst.multi INFO: found job 16912323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 175 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2227c802e71b20480100f7e665de45f with result from task 175 parsl.dataflow.dflow DEBUG: Task 175 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 176 has memoization hash 4dcb719d08ee950eec049d6677d4c060 parsl.dataflow.memoization INFO: Task 176 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46815940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 176 try 0 launched on executor multi with executor id 1035 parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/4c23a0e8-0b53-47ea-b339-66271b6da505_calibrate_855043_59.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/4c23a0e8-0b53-47ea-b339-66271b6da505_calibrate_855043_59.stderr parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/ee502b78-df1d-4e10-9bfb-c08e2fc994d9_characterizeImage_855043_59.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/ee502b78-df1d-4e10-9bfb-c08e2fc994d9_characterizeImage_855043_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:54 proc_lsst.multi:146 [INFO] found job 16912324 in provider escience_a40 proc_lsst.multi INFO: found job 16912324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 638 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e87edd3f163b21e01289bd7dcb37e0d with result from task 638 parsl.dataflow.dflow DEBUG: Task 638 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/70b426ed-d278-41e9-9939-6423819e32f7_calibrate_855044_50.stdout parsl.dataflow.dflow INFO: Standard error for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/70b426ed-d278-41e9-9939-6423819e32f7_calibrate_855044_50.stderr parsl.dataflow.dflow INFO: Task 959 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a075323ff66790f17be297a9a490de9 with result from task 959 parsl.dataflow.dflow DEBUG: Task 959 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/b9a96252-f87d-4ab2-836e-75b73ff408bb_calibrate_855137_22.stdout parsl.dataflow.dflow INFO: Standard error for task 959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/b9a96252-f87d-4ab2-836e-75b73ff408bb_calibrate_855137_22.stderr parsl.dataflow.dflow INFO: Task 1045 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 35ea2bf0db1d80e5cdac3c6350ea1a91 with result from task 1045 parsl.dataflow.dflow DEBUG: Task 1045 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1046 has memoization hash 2108c9061f80894aa6d1989b23e2766b parsl.dataflow.memoization INFO: Task 1046 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45543740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1046 try 0 launched on executor multi with executor id 1036 parsl.dataflow.dflow INFO: Standard output for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/7233f557-5c3a-4bbc-9f1a-3d62e748480d_calibrate_855044_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/7233f557-5c3a-4bbc-9f1a-3d62e748480d_calibrate_855044_54.stderr parsl.dataflow.dflow INFO: Standard output for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/b6a8fa87-aa73-4766-ab7f-2140139ee391_characterizeImage_855044_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/b6a8fa87-aa73-4766-ab7f-2140139ee391_characterizeImage_855044_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:55 proc_lsst.multi:146 [INFO] found job 16912325 in provider escience_a40 proc_lsst.multi INFO: found job 16912325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 485 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 258f33fc2fe25dcaf3e4f2b691a5c9c2 with result from task 485 parsl.dataflow.dflow DEBUG: Task 485 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/1950b431-c469-4808-8043-19acbd8b7ada_calibrate_855136_27.stdout parsl.dataflow.dflow INFO: Standard error for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/1950b431-c469-4808-8043-19acbd8b7ada_calibrate_855136_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:55 proc_lsst.multi:146 [INFO] found job 16912326 in provider escience_a40 proc_lsst.multi INFO: found job 16912326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 472 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8764c1310a5dafd83418203cd29024b with result from task 472 parsl.dataflow.dflow DEBUG: Task 472 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 473 has memoization hash a3b60a893efb26018bb5d7e711b42da7 parsl.dataflow.memoization INFO: Task 473 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456cf060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 473 try 0 launched on executor multi with executor id 1037 parsl.dataflow.dflow INFO: Standard output for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/35ec5dc2-99d7-4484-b545-da5c7642a799_calibrate_855042_42.stdout parsl.dataflow.dflow INFO: Standard error for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/35ec5dc2-99d7-4484-b545-da5c7642a799_calibrate_855042_42.stderr parsl.dataflow.dflow INFO: Standard output for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/a745721b-67a8-4b69-842d-e03d5a48820c_characterizeImage_855042_42.stdout parsl.dataflow.dflow INFO: Standard error for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/a745721b-67a8-4b69-842d-e03d5a48820c_characterizeImage_855042_42.stderr parsl.dataflow.dflow INFO: Task 287 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66df182438cc20212da5d4b36952c838 with result from task 287 parsl.dataflow.dflow DEBUG: Task 287 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/698e6f7d-c2e9-49f5-b39a-8bdef912dd9d_calibrate_855137_29.stdout parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/698e6f7d-c2e9-49f5-b39a-8bdef912dd9d_calibrate_855137_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:55 proc_lsst.multi:146 [INFO] found job 16912327 in provider escience_a40 proc_lsst.multi INFO: found job 16912327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 173 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0433bc892be1f46d75b7f3c759408521 with result from task 173 parsl.dataflow.dflow DEBUG: Task 173 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/e6020d81-39fb-4fc7-a747-51346446508a_calibrate_855138_49.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/e6020d81-39fb-4fc7-a747-51346446508a_calibrate_855138_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:55 proc_lsst.multi:146 [INFO] found job 16912328 in provider escience_a40 proc_lsst.multi INFO: found job 16912328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 725 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e67f4e4fa3fe4be88db42d9520a5dc0 with result from task 725 parsl.dataflow.dflow DEBUG: Task 725 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/628228aa-6b1f-4426-85a1-6b6bfcf806a9_calibrate_855138_12.stdout parsl.dataflow.dflow INFO: Standard error for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/628228aa-6b1f-4426-85a1-6b6bfcf806a9_calibrate_855138_12.stderr parsl.dataflow.dflow INFO: Task 452 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d250e93a816b13c6939d35ffb52c515d with result from task 452 parsl.dataflow.dflow DEBUG: Task 452 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/c1d3eb70-34ea-463d-8305-b54f531508ad_calibrate_855138_9.stdout parsl.dataflow.dflow INFO: Standard error for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/c1d3eb70-34ea-463d-8305-b54f531508ad_calibrate_855138_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:56 proc_lsst.multi:146 [INFO] found job 16912329 in provider escience_a40 proc_lsst.multi INFO: found job 16912329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:56 proc_lsst.multi:146 [INFO] found job 16912330 in provider escience_a40 proc_lsst.multi INFO: found job 16912330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:56 proc_lsst.multi:146 [INFO] found job 16912331 in provider escience_a40 proc_lsst.multi INFO: found job 16912331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 644 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53d9738db3183bf827e05916062f133b with result from task 644 parsl.dataflow.dflow DEBUG: Task 644 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/f1f98230-728f-4d9e-b00b-f7c48b76b32e_calibrate_855043_33.stdout parsl.dataflow.dflow INFO: Standard error for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/f1f98230-728f-4d9e-b00b-f7c48b76b32e_calibrate_855043_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:56 proc_lsst.multi:146 [INFO] found job 16912332 in provider escience_a40 proc_lsst.multi INFO: found job 16912332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:57 proc_lsst.multi:146 [INFO] found job 16912333 in provider escience_a40 proc_lsst.multi INFO: found job 16912333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 682 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c02d8945968feac74ddadbe302a0e756 with result from task 682 parsl.dataflow.dflow DEBUG: Task 682 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 683 has memoization hash 26d3cd7c8c12a3cc3f0aa0ae67fae159 parsl.dataflow.memoization INFO: Task 683 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 683 try 0 launched on executor multi with executor id 1038 parsl.dataflow.dflow INFO: Standard output for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/87745ffd-de8e-41ce-9ef3-cf7c45053892_calibrate_855043_28.stdout parsl.dataflow.dflow INFO: Standard error for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/87745ffd-de8e-41ce-9ef3-cf7c45053892_calibrate_855043_28.stderr parsl.dataflow.dflow INFO: Standard output for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/eb792d5b-e5a4-43ca-8d0d-3d614ff10633_characterizeImage_855043_28.stdout parsl.dataflow.dflow INFO: Standard error for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/eb792d5b-e5a4-43ca-8d0d-3d614ff10633_characterizeImage_855043_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:57 proc_lsst.multi:146 [INFO] found job 16912334 in provider escience_a40 proc_lsst.multi INFO: found job 16912334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 215 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5b5eba8c032672117f1d561c12c8583 with result from task 215 parsl.dataflow.dflow DEBUG: Task 215 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/fdbd57a6-7884-445c-bd5d-839b42feabba_calibrate_855043_12.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/fdbd57a6-7884-445c-bd5d-839b42feabba_calibrate_855043_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:57 proc_lsst.multi:146 [INFO] found job 16912335 in provider escience_a40 proc_lsst.multi INFO: found job 16912335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 823 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 54acd9c9fc4c11a5546d07ff78455f42 with result from task 823 parsl.dataflow.dflow DEBUG: Task 823 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 824 has memoization hash 3790f0a75984edc191f16d47764acdd9 parsl.dataflow.memoization INFO: Task 824 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456642c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 824 try 0 launched on executor multi with executor id 1039 parsl.dataflow.dflow INFO: Standard output for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/76c63b73-6669-4a84-a5e7-b55aabf04b86_calibrate_855137_38.stdout parsl.dataflow.dflow INFO: Standard error for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/76c63b73-6669-4a84-a5e7-b55aabf04b86_calibrate_855137_38.stderr parsl.dataflow.dflow INFO: Standard output for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/7a81f2a9-2a79-48b4-b095-477728d406a6_characterizeImage_855137_38.stdout parsl.dataflow.dflow INFO: Standard error for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/7a81f2a9-2a79-48b4-b095-477728d406a6_characterizeImage_855137_38.stderr parsl.dataflow.dflow INFO: Task 209 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c03cf38e53517d318be14161c3a0254 with result from task 209 parsl.dataflow.dflow DEBUG: Task 209 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/e4671f0f-5fc2-412b-afa2-61ab06b56ee2_calibrate_855042_17.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/e4671f0f-5fc2-412b-afa2-61ab06b56ee2_calibrate_855042_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:58 proc_lsst.multi:146 [INFO] found job 16912336 in provider escience_a40 proc_lsst.multi INFO: found job 16912336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 685 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4250d8da97c588b38a6b80dc925ba1b1 with result from task 685 parsl.dataflow.dflow DEBUG: Task 685 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 686 has memoization hash 210e3f43bd6c568e03b26b51151e39f7 parsl.dataflow.memoization INFO: Task 686 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 686 try 0 launched on executor multi with executor id 1040 parsl.dataflow.dflow INFO: Standard output for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/977704e8-a548-4b92-a606-e81a5d29db4a_calibrate_855044_52.stdout parsl.dataflow.dflow INFO: Standard error for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/977704e8-a548-4b92-a606-e81a5d29db4a_calibrate_855044_52.stderr parsl.dataflow.dflow INFO: Standard output for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/0f0888ee-101b-41ea-b843-fe9d2e5169a9_characterizeImage_855044_52.stdout parsl.dataflow.dflow INFO: Standard error for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/0f0888ee-101b-41ea-b843-fe9d2e5169a9_characterizeImage_855044_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:58 proc_lsst.multi:146 [INFO] found job 16912337 in provider escience_a40 proc_lsst.multi INFO: found job 16912337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 1096 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d066c048630182efa29b5aba12770be0 with result from task 1096 parsl.dataflow.dflow DEBUG: Task 1096 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1097 has memoization hash 6423e5287943954fbe03549310a439b9 parsl.dataflow.memoization INFO: Task 1097 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45391440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1097 try 0 launched on executor multi with executor id 1041 parsl.dataflow.dflow INFO: Standard output for task 1097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/a21e61c4-ebac-4c0a-8e59-765180522dd7_calibrate_855137_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/a21e61c4-ebac-4c0a-8e59-765180522dd7_calibrate_855137_37.stderr parsl.dataflow.dflow INFO: Standard output for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/06d25411-0ce6-40dc-a558-acc5bdf14a39_characterizeImage_855137_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/06d25411-0ce6-40dc-a558-acc5bdf14a39_characterizeImage_855137_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:58 proc_lsst.multi:146 [INFO] found job 16912338 in provider checkpoint proc_lsst.multi INFO: found job 16912338 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 715 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 609d67b52a9a201a4d262247f40d532c with result from task 715 parsl.dataflow.dflow DEBUG: Task 715 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 716 has memoization hash 8fcc5ccf9c31d4dc34a1af73783e38bd parsl.dataflow.memoization INFO: Task 716 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d0fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 716 try 0 launched on executor multi with executor id 1042 parsl.dataflow.dflow INFO: Standard output for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/760598ab-0dd3-49a7-aee6-49f159364ebe_calibrate_855137_12.stdout parsl.dataflow.dflow INFO: Standard error for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/760598ab-0dd3-49a7-aee6-49f159364ebe_calibrate_855137_12.stderr parsl.dataflow.dflow INFO: Standard output for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/31a9a9a9-ebfd-4fb0-a3ab-c43e98b3cb1e_characterizeImage_855137_12.stdout parsl.dataflow.dflow INFO: Standard error for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/31a9a9a9-ebfd-4fb0-a3ab-c43e98b3cb1e_characterizeImage_855137_12.stderr parsl.dataflow.dflow INFO: Task 962 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f7301f9d693f83f0aaf595d79c4c00f with result from task 962 parsl.dataflow.dflow DEBUG: Task 962 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/a7c8b557-db6b-4836-a5a0-ea61fb5992f8_calibrate_855043_27.stdout parsl.dataflow.dflow INFO: Standard error for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/a7c8b557-db6b-4836-a5a0-ea61fb5992f8_calibrate_855043_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:58 proc_lsst.multi:146 [INFO] found job 16912339 in provider checkpoint proc_lsst.multi INFO: found job 16912339 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 911 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e392534cf53919d702d4fc88a3b66a7 with result from task 911 parsl.dataflow.dflow DEBUG: Task 911 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/d9937d5d-a352-4c6f-87ce-3aece9e63183_calibrate_855043_8.stdout parsl.dataflow.dflow INFO: Standard error for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/d9937d5d-a352-4c6f-87ce-3aece9e63183_calibrate_855043_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:59 proc_lsst.multi:146 [INFO] found job 16912340 in provider checkpoint proc_lsst.multi INFO: found job 16912340 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 718 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6f8d768665ddaeca389b8156a17b941e with result from task 718 parsl.dataflow.dflow DEBUG: Task 718 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 719 has memoization hash d6fdc18a5a55d019f81da244436fb35b parsl.dataflow.memoization INFO: Task 719 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d11c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 719 try 0 launched on executor multi with executor id 1043 parsl.dataflow.dflow INFO: Standard output for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/4f2a0011-b562-4a4e-b59d-89c419d876d3_calibrate_855042_6.stdout parsl.dataflow.dflow INFO: Standard error for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/4f2a0011-b562-4a4e-b59d-89c419d876d3_calibrate_855042_6.stderr parsl.dataflow.dflow INFO: Standard output for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/c9f38fef-ae51-4422-8a5b-af51f4a87389_characterizeImage_855042_6.stdout parsl.dataflow.dflow INFO: Standard error for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/c9f38fef-ae51-4422-8a5b-af51f4a87389_characterizeImage_855042_6.stderr parsl.dataflow.dflow INFO: Task 950 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2ae92a5e8d4e0854e259668a19c96ab with result from task 950 parsl.dataflow.dflow DEBUG: Task 950 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/451b17f0-ad0c-418f-a833-f10d0035d841_calibrate_855136_58.stdout parsl.dataflow.dflow INFO: Standard error for task 950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/451b17f0-ad0c-418f-a833-f10d0035d841_calibrate_855136_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:59 proc_lsst.multi:146 [INFO] found job 16912341 in provider checkpoint proc_lsst.multi INFO: found job 16912341 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 952 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 34d178d8ff3492b19f8e9af80b1bd71f with result from task 952 parsl.dataflow.dflow DEBUG: Task 952 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 953 has memoization hash c3a599732bdf3c2ade51ecda5c66ca08 parsl.dataflow.memoization INFO: Task 953 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454fa660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 953 try 0 launched on executor multi with executor id 1044 parsl.dataflow.dflow INFO: Standard output for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/2aa448c5-a320-42fa-a807-548b1814fcb8_calibrate_855042_11.stdout parsl.dataflow.dflow INFO: Standard error for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/2aa448c5-a320-42fa-a807-548b1814fcb8_calibrate_855042_11.stderr parsl.dataflow.dflow INFO: Standard output for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/f042c185-8882-44a5-bf17-2f304c8bbe54_characterizeImage_855042_11.stdout parsl.dataflow.dflow INFO: Standard error for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/f042c185-8882-44a5-bf17-2f304c8bbe54_characterizeImage_855042_11.stderr parsl.dataflow.dflow INFO: Task 254 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49466d15d5289b57fb35c496cbd2ca62 with result from task 254 parsl.dataflow.dflow DEBUG: Task 254 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/36cce5b1-6ef2-45e7-b06b-52d089636224_calibrate_855043_6.stdout parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/36cce5b1-6ef2-45e7-b06b-52d089636224_calibrate_855043_6.stderr parsl.dataflow.dflow INFO: Task 397 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50fc7b39947cc2dd6d87e14eb24034ca with result from task 397 parsl.dataflow.dflow DEBUG: Task 397 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 398 has memoization hash 4212eb15d5fbba9654c9196e64610f6d parsl.dataflow.memoization INFO: Task 398 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4687ab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 398 try 0 launched on executor multi with executor id 1045 parsl.dataflow.dflow INFO: Standard output for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/09dd010a-cc95-40b2-9f42-6f7f4e2643ef_calibrate_855138_48.stdout parsl.dataflow.dflow INFO: Standard error for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/09dd010a-cc95-40b2-9f42-6f7f4e2643ef_calibrate_855138_48.stderr parsl.dataflow.dflow INFO: Standard output for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/a78d370b-f3a7-45b8-ba3d-74d7f5c88a21_characterizeImage_855138_48.stdout parsl.dataflow.dflow INFO: Standard error for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/a78d370b-f3a7-45b8-ba3d-74d7f5c88a21_characterizeImage_855138_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:59 proc_lsst.multi:146 [INFO] found job 16912342 in provider checkpoint proc_lsst.multi INFO: found job 16912342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:19:59 proc_lsst.multi:146 [INFO] found job 16912343 in provider checkpoint proc_lsst.multi INFO: found job 16912343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 1099 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a390ed2e4656a37563152dc783c02716 with result from task 1099 parsl.dataflow.dflow DEBUG: Task 1099 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1100 has memoization hash 91dfa564aab0811ea8cc4599e94146de parsl.dataflow.memoization INFO: Task 1100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45391800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1100 try 0 launched on executor multi with executor id 1046 parsl.dataflow.dflow INFO: Standard output for task 1100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/8dbfe311-2339-4015-8464-a144f2c8f5f3_calibrate_855042_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/8dbfe311-2339-4015-8464-a144f2c8f5f3_calibrate_855042_46.stderr parsl.dataflow.dflow INFO: Standard output for task 1099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/e5d5efe8-cbb8-44fd-b5bc-1c7849807054_characterizeImage_855042_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/e5d5efe8-cbb8-44fd-b5bc-1c7849807054_characterizeImage_855042_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:00 proc_lsst.multi:146 [INFO] found job 16912344 in provider checkpoint proc_lsst.multi INFO: found job 16912344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:00 proc_lsst.multi:146 [INFO] found job 16912345 in provider checkpoint proc_lsst.multi INFO: found job 16912345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:00 proc_lsst.multi:146 [INFO] found job 16912346 in provider checkpoint proc_lsst.multi INFO: found job 16912346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 194 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2e63d903bde3d4d65f2f238d86db5f8 with result from task 194 parsl.dataflow.dflow DEBUG: Task 194 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/147501f0-f2cb-4d87-a1a5-5c0d45d65d97_calibrate_855136_33.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/147501f0-f2cb-4d87-a1a5-5c0d45d65d97_calibrate_855136_33.stderr parsl.dataflow.dflow INFO: Task 926 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b888f4c9bbe884b2e598f366dd516641 with result from task 926 parsl.dataflow.dflow DEBUG: Task 926 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/d5149127-61cc-4977-b9bb-f101b5823d14_calibrate_855138_46.stdout parsl.dataflow.dflow INFO: Standard error for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/d5149127-61cc-4977-b9bb-f101b5823d14_calibrate_855138_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:00 proc_lsst.multi:146 [INFO] found job 16912347 in provider checkpoint proc_lsst.multi INFO: found job 16912347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:01 proc_lsst.multi:146 [INFO] found job 16912348 in provider checkpoint proc_lsst.multi INFO: found job 16912348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:01 proc_lsst.multi:146 [INFO] found job 16912349 in provider checkpoint proc_lsst.multi INFO: found job 16912349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:01 proc_lsst.multi:146 [INFO] found job 16912350 in provider checkpoint proc_lsst.multi INFO: found job 16912350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:01 proc_lsst.multi:146 [INFO] found job 16912351 in provider checkpoint proc_lsst.multi INFO: found job 16912351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 1027 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c386a321548e15810448d85624868210 with result from task 1027 parsl.dataflow.dflow DEBUG: Task 1027 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1028 has memoization hash 2033536d3aae7907687bba4833cbdf7a parsl.dataflow.memoization INFO: Task 1028 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45541580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1028 try 0 launched on executor multi with executor id 1047 parsl.dataflow.dflow INFO: Standard output for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/face6c96-4541-4960-82d3-60560ad8ade3_calibrate_855042_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/face6c96-4541-4960-82d3-60560ad8ade3_calibrate_855042_56.stderr parsl.dataflow.dflow INFO: Standard output for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/03b39fdf-fa97-4709-ad6b-24d8feb4e95e_characterizeImage_855042_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/03b39fdf-fa97-4709-ad6b-24d8feb4e95e_characterizeImage_855042_56.stderr parsl.dataflow.dflow INFO: Task 532 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17dfec860bc0662f83dceb23492de9e0 with result from task 532 parsl.dataflow.dflow DEBUG: Task 532 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 533 has memoization hash 8aafeadf358ddc433f4522b36d110dc8 parsl.dataflow.memoization INFO: Task 533 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45703b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 533 try 0 launched on executor multi with executor id 1048 parsl.dataflow.dflow INFO: Standard output for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/cff4ad65-c038-4e48-87b7-ad5a26b2780d_calibrate_855042_32.stdout parsl.dataflow.dflow INFO: Standard error for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/cff4ad65-c038-4e48-87b7-ad5a26b2780d_calibrate_855042_32.stderr parsl.dataflow.dflow INFO: Standard output for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/4cb5ddd4-8f4d-4a37-a6c2-6ba6b178ffdc_characterizeImage_855042_32.stdout parsl.dataflow.dflow INFO: Standard error for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/4cb5ddd4-8f4d-4a37-a6c2-6ba6b178ffdc_characterizeImage_855042_32.stderr parsl.dataflow.dflow INFO: Task 455 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b99bde04c8ed542b8727fee68b6c5e6 with result from task 455 parsl.dataflow.dflow DEBUG: Task 455 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/ea03c0c0-da17-4c8c-a881-2187f74efe12_calibrate_855137_20.stdout parsl.dataflow.dflow INFO: Standard error for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/ea03c0c0-da17-4c8c-a881-2187f74efe12_calibrate_855137_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:02 proc_lsst.multi:146 [INFO] found job 16912352 in provider checkpoint proc_lsst.multi INFO: found job 16912352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:02 proc_lsst.multi:146 [INFO] found job 16912353 in provider checkpoint proc_lsst.multi INFO: found job 16912353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 227 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e2399e2b58072ee79dabbe1eee80fe1 with result from task 227 parsl.dataflow.dflow DEBUG: Task 227 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/f6ddf3a4-6fdc-49cf-b96c-e252f3e6faae_calibrate_855137_48.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/f6ddf3a4-6fdc-49cf-b96c-e252f3e6faae_calibrate_855137_48.stderr parsl.dataflow.dflow INFO: Task 1081 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e07cf5391f82f96308bed92916e4bb0a with result from task 1081 parsl.dataflow.dflow DEBUG: Task 1081 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1082 has memoization hash 107ad66928009f7aa06f0c2c087a53e7 parsl.dataflow.memoization INFO: Task 1082 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa453904a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1082 try 0 launched on executor multi with executor id 1049 parsl.dataflow.dflow INFO: Standard output for task 1082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/9a60981d-36a0-415e-a405-a0693a27b647_calibrate_855043_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/9a60981d-36a0-415e-a405-a0693a27b647_calibrate_855043_22.stderr parsl.dataflow.dflow INFO: Standard output for task 1081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/dead2eb3-23ef-4210-80df-eef151c41d4c_characterizeImage_855043_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/dead2eb3-23ef-4210-80df-eef151c41d4c_characterizeImage_855043_22.stderr parsl.dataflow.dflow INFO: Task 325 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b3533700f85923ab6301c0a2d518357 with result from task 325 parsl.dataflow.dflow DEBUG: Task 325 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 326 has memoization hash a8965267c5f84770f82434360901d319 parsl.dataflow.memoization INFO: Task 326 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 326 try 0 launched on executor multi with executor id 1050 parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/b11fb852-8434-4b46-9693-a241b49adaa3_calibrate_855042_52.stdout parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/b11fb852-8434-4b46-9693-a241b49adaa3_calibrate_855042_52.stderr parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/7828124f-5a80-4b84-9b93-af0c1b96b9bd_characterizeImage_855042_52.stdout parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/7828124f-5a80-4b84-9b93-af0c1b96b9bd_characterizeImage_855042_52.stderr parsl.dataflow.dflow INFO: Task 1063 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 30e9aa5d672cdf61120c8f55d1f55ed5 with result from task 1063 parsl.dataflow.dflow DEBUG: Task 1063 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1064 has memoization hash 016a18c3e972dbafa52b077d0f70dd54 parsl.dataflow.memoization INFO: Task 1064 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45540680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1064 try 0 launched on executor multi with executor id 1051 parsl.dataflow.dflow INFO: Standard output for task 1064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/b7d9b68d-497f-4ba6-998e-cfb63289d688_calibrate_855044_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/b7d9b68d-497f-4ba6-998e-cfb63289d688_calibrate_855044_19.stderr parsl.dataflow.dflow INFO: Standard output for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/17b5038d-27ad-4408-86e8-c82832441379_characterizeImage_855044_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/17b5038d-27ad-4408-86e8-c82832441379_characterizeImage_855044_19.stderr parsl.dataflow.dflow INFO: Task 976 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 117ca0fb41140ea06470679fc7e62a8e with result from task 976 parsl.dataflow.dflow DEBUG: Task 976 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 977 has memoization hash 1480303512b8c4b9e4e7718a349f4f02 parsl.dataflow.memoization INFO: Task 977 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454f9080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 977 try 0 launched on executor multi with executor id 1052 parsl.dataflow.dflow INFO: Standard output for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/bab9dc8e-91bc-4987-888a-29113d3c1826_calibrate_855044_59.stdout parsl.dataflow.dflow INFO: Standard error for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/bab9dc8e-91bc-4987-888a-29113d3c1826_calibrate_855044_59.stderr parsl.dataflow.dflow INFO: Standard output for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/f2d8a9c6-9b7f-4b59-872b-0ebedd4fe87b_characterizeImage_855044_59.stdout parsl.dataflow.dflow INFO: Standard error for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/f2d8a9c6-9b7f-4b59-872b-0ebedd4fe87b_characterizeImage_855044_59.stderr parsl.dataflow.dflow INFO: Task 940 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d8a4739a7907782958ac5c3bc42a497 with result from task 940 parsl.dataflow.dflow DEBUG: Task 940 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 941 has memoization hash d0346757f6621fdbb80629da6464fc0d parsl.dataflow.memoization INFO: Task 941 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4549b740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 941 try 0 launched on executor multi with executor id 1053 parsl.dataflow.dflow INFO: Standard output for task 941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/d2940fa8-254e-4bdf-a822-0b03f1cf2cee_calibrate_855138_30.stdout parsl.dataflow.dflow INFO: Standard error for task 941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/d2940fa8-254e-4bdf-a822-0b03f1cf2cee_calibrate_855138_30.stderr parsl.dataflow.dflow INFO: Standard output for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/2a0661d8-8ed5-4449-906e-10c5822d43da_characterizeImage_855138_30.stdout parsl.dataflow.dflow INFO: Standard error for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/2a0661d8-8ed5-4449-906e-10c5822d43da_characterizeImage_855138_30.stderr parsl.dataflow.dflow INFO: Task 47 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 118a473971fe6ca82da4bdbf54320dc7 with result from task 47 parsl.dataflow.dflow DEBUG: Task 47 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/3cc55836-d648-4972-b6ac-b9e6f647900d_calibrate_855138_32.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/3cc55836-d648-4972-b6ac-b9e6f647900d_calibrate_855138_32.stderr parsl.dataflow.dflow INFO: Task 200 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6f40c6645e075c2756b87d71feb91ccf with result from task 200 parsl.dataflow.dflow DEBUG: Task 200 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/512b77a2-c861-46a7-839b-49ade36d2b2f_calibrate_855042_10.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/512b77a2-c861-46a7-839b-49ade36d2b2f_calibrate_855042_10.stderr parsl.dataflow.dflow INFO: Task 217 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 444ad249247a24837c88048c1f89fb17 with result from task 217 parsl.dataflow.dflow DEBUG: Task 217 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 218 has memoization hash d84a293737d18b4d8d632d2933969b01 parsl.dataflow.memoization INFO: Task 218 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46834540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 218 try 0 launched on executor multi with executor id 1054 parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/4a798e90-2b57-48bc-82e3-60f42159d00d_calibrate_855043_17.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/4a798e90-2b57-48bc-82e3-60f42159d00d_calibrate_855043_17.stderr parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/acc6d99f-747d-4c26-9479-29b9904d49b5_characterizeImage_855043_17.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/acc6d99f-747d-4c26-9479-29b9904d49b5_characterizeImage_855043_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:04 proc_lsst.multi:146 [INFO] found job 16912354 in provider checkpoint proc_lsst.multi INFO: found job 16912354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 500 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92fd4a8545d2443465f79dd730fdaef4 with result from task 500 parsl.dataflow.dflow DEBUG: Task 500 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/40a86c7d-dd94-4b9b-b59b-e5ee20411a8d_calibrate_855137_28.stdout parsl.dataflow.dflow INFO: Standard error for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/40a86c7d-dd94-4b9b-b59b-e5ee20411a8d_calibrate_855137_28.stderr parsl.dataflow.dflow INFO: Task 721 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1ebd70dcfed6b4f9ceee399d2f1079d with result from task 721 parsl.dataflow.dflow DEBUG: Task 721 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 722 has memoization hash b314f6b7ca035edad7859d4827822b98 parsl.dataflow.memoization INFO: Task 722 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d1620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 722 try 0 launched on executor multi with executor id 1055 parsl.dataflow.dflow INFO: Standard output for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/d7df3f39-814a-48a4-b2df-5039671ab585_calibrate_855044_9.stdout parsl.dataflow.dflow INFO: Standard error for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/d7df3f39-814a-48a4-b2df-5039671ab585_calibrate_855044_9.stderr parsl.dataflow.dflow INFO: Standard output for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/31732b93-acf4-4eac-bc11-a37675124968_characterizeImage_855044_9.stdout parsl.dataflow.dflow INFO: Standard error for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/31732b93-acf4-4eac-bc11-a37675124968_characterizeImage_855044_9.stderr parsl.dataflow.dflow INFO: Task 997 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9dac8b9774aec2248a551d3ce8ac0407 with result from task 997 parsl.dataflow.dflow DEBUG: Task 997 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 998 has memoization hash 828ba53f30de359cc4d989fd9be10e2b parsl.dataflow.memoization INFO: Task 998 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454fb880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 998 try 0 launched on executor multi with executor id 1056 parsl.dataflow.dflow INFO: Standard output for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/63ab6e21-f386-421f-af25-68be56d5945e_calibrate_855138_26.stdout parsl.dataflow.dflow INFO: Standard error for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/63ab6e21-f386-421f-af25-68be56d5945e_calibrate_855138_26.stderr parsl.dataflow.dflow INFO: Standard output for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/3b7f08a6-6510-4c13-85fb-a7ca4488e4dd_characterizeImage_855138_26.stdout parsl.dataflow.dflow INFO: Standard error for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/3b7f08a6-6510-4c13-85fb-a7ca4488e4dd_characterizeImage_855138_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:04 proc_lsst.multi:146 [INFO] found job 16912355 in provider checkpoint proc_lsst.multi INFO: found job 16912355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 848 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 19b5ae5a837423b362f7dc35c78e2e39 with result from task 848 parsl.dataflow.dflow DEBUG: Task 848 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/3cc80c49-d469-410b-aca4-de6e72971a0c_calibrate_855044_7.stdout parsl.dataflow.dflow INFO: Standard error for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/3cc80c49-d469-410b-aca4-de6e72971a0c_calibrate_855044_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:05 proc_lsst.multi:146 [INFO] found job 16912356 in provider checkpoint proc_lsst.multi INFO: found job 16912356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 712 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f75048fe2dcc84b871bfacf5324a5817 with result from task 712 parsl.dataflow.dflow DEBUG: Task 712 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 713 has memoization hash 3321a09d26f1238bed3bce5f0f8e95f5 parsl.dataflow.memoization INFO: Task 713 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455d0ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 713 try 0 launched on executor multi with executor id 1057 parsl.dataflow.dflow INFO: Standard output for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/d40504f6-7dc0-4a56-8ade-788d1b3dc53e_calibrate_855138_27.stdout parsl.dataflow.dflow INFO: Standard error for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/d40504f6-7dc0-4a56-8ade-788d1b3dc53e_calibrate_855138_27.stderr parsl.dataflow.dflow INFO: Standard output for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/5f30893a-935f-41dc-9447-6fb33ccac56e_characterizeImage_855138_27.stdout parsl.dataflow.dflow INFO: Standard error for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/5f30893a-935f-41dc-9447-6fb33ccac56e_characterizeImage_855138_27.stderr parsl.dataflow.dflow INFO: Task 569 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a26e877d4fc22006e6efe3b103d4f2c5 with result from task 569 parsl.dataflow.dflow DEBUG: Task 569 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/976ba8fe-13f2-4bc0-af97-a129a85e64c7_calibrate_855043_50.stdout parsl.dataflow.dflow INFO: Standard error for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/976ba8fe-13f2-4bc0-af97-a129a85e64c7_calibrate_855043_50.stderr parsl.dataflow.dflow INFO: Task 857 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97f1fcfa46a8903f19373dfb7254bffb with result from task 857 parsl.dataflow.dflow DEBUG: Task 857 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/8acf325b-4e50-4a67-80a8-fa72b454d897_calibrate_855043_43.stdout parsl.dataflow.dflow INFO: Standard error for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/8acf325b-4e50-4a67-80a8-fa72b454d897_calibrate_855043_43.stderr parsl.dataflow.dflow INFO: Task 263 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0e7bcd43e58a9b11ed685059b93dd3c with result from task 263 parsl.dataflow.dflow DEBUG: Task 263 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/eebcc2a4-e110-45e5-a603-867bf34db737_calibrate_855136_21.stdout parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/eebcc2a4-e110-45e5-a603-867bf34db737_calibrate_855136_21.stderr parsl.dataflow.dflow INFO: Task 230 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73a47c5805e1f7190a0467ef0822445b with result from task 230 parsl.dataflow.dflow DEBUG: Task 230 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/08f565e1-117d-40d9-a9d9-ccf131555b50_calibrate_855137_32.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/08f565e1-117d-40d9-a9d9-ccf131555b50_calibrate_855137_32.stderr parsl.dataflow.dflow INFO: Task 1015 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09598be12c1d3c26e5c92fa26bc52474 with result from task 1015 parsl.dataflow.dflow DEBUG: Task 1015 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1016 has memoization hash a03452a6d415da69349b496e1e6cbea3 parsl.dataflow.memoization INFO: Task 1016 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45540900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1016 try 0 launched on executor multi with executor id 1058 parsl.dataflow.dflow INFO: Standard output for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/58ccb9cf-db71-4799-9d17-0420bc7d247f_calibrate_855138_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/58ccb9cf-db71-4799-9d17-0420bc7d247f_calibrate_855138_52.stderr parsl.dataflow.dflow INFO: Standard output for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/81857641-7f3a-421c-994d-05279ac99483_characterizeImage_855138_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/81857641-7f3a-421c-994d-05279ac99483_characterizeImage_855138_52.stderr parsl.dataflow.dflow INFO: Task 796 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 977ee046be8ec24e8fa23f3bb2a4d00c with result from task 796 parsl.dataflow.dflow DEBUG: Task 796 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 797 has memoization hash 5e65751d42d3c4bf03077ffb2d4cb0b2 parsl.dataflow.memoization INFO: Task 797 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 797 try 0 launched on executor multi with executor id 1059 parsl.dataflow.dflow INFO: Standard output for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/f7a34bf7-38c7-42fc-a00e-c0b47d4818b4_calibrate_855136_39.stdout parsl.dataflow.dflow INFO: Standard error for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/f7a34bf7-38c7-42fc-a00e-c0b47d4818b4_calibrate_855136_39.stderr parsl.dataflow.dflow INFO: Standard output for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/9144c2da-e3b9-44d0-9230-edb4da76e6dd_characterizeImage_855136_39.stdout parsl.dataflow.dflow INFO: Standard error for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/9144c2da-e3b9-44d0-9230-edb4da76e6dd_characterizeImage_855136_39.stderr parsl.dataflow.dflow INFO: Task 656 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5346a4fd026de12a3bffebbb9a3dea22 with result from task 656 parsl.dataflow.dflow DEBUG: Task 656 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/ca50a906-23ee-46dc-847a-67558d460985_calibrate_855043_39.stdout parsl.dataflow.dflow INFO: Standard error for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/ca50a906-23ee-46dc-847a-67558d460985_calibrate_855043_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:07 proc_lsst.multi:146 [INFO] found job 16912359 in provider checkpoint proc_lsst.multi INFO: found job 16912359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:07 proc_lsst.multi:146 [INFO] found job 16912360 in provider checkpoint proc_lsst.multi INFO: found job 16912360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 631 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e784c2eca78637ac23981f726aff79b4 with result from task 631 parsl.dataflow.dflow DEBUG: Task 631 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 632 has memoization hash dee5d221b908d65e4af8765a4ffc0682 parsl.dataflow.memoization INFO: Task 632 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa457439c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 632 try 0 launched on executor multi with executor id 1060 parsl.dataflow.dflow INFO: Standard output for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/fb77bb20-b71d-4460-aa25-ae8900e1b211_calibrate_855044_30.stdout parsl.dataflow.dflow INFO: Standard error for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/fb77bb20-b71d-4460-aa25-ae8900e1b211_calibrate_855044_30.stderr parsl.dataflow.dflow INFO: Standard output for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/8af16452-4b58-4109-894e-0f033e6faafc_characterizeImage_855044_30.stdout parsl.dataflow.dflow INFO: Standard error for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/8af16452-4b58-4109-894e-0f033e6faafc_characterizeImage_855044_30.stderr parsl.dataflow.dflow INFO: Task 203 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ebe17a92e0f52eb467ca3ce90fea5a2e with result from task 203 parsl.dataflow.dflow DEBUG: Task 203 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/eef30ba1-32e2-4fde-a71e-7d6c53a6c420_calibrate_855138_33.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/eef30ba1-32e2-4fde-a71e-7d6c53a6c420_calibrate_855138_33.stderr parsl.dataflow.dflow INFO: Task 347 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5a4b86d30e4e07f32e9c524adbb792f with result from task 347 parsl.dataflow.dflow DEBUG: Task 347 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/ff1d6b49-fd3f-4969-80a4-d085692f6d38_calibrate_855043_25.stdout parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/ff1d6b49-fd3f-4969-80a4-d085692f6d38_calibrate_855043_25.stderr parsl.dataflow.dflow INFO: Task 320 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d9df48ea8ec1946b26b06e86cc20e2c with result from task 320 parsl.dataflow.dflow DEBUG: Task 320 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/ff183dc0-60ac-4295-a81f-ab7573aacc95_calibrate_855044_56.stdout parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/ff183dc0-60ac-4295-a81f-ab7573aacc95_calibrate_855044_56.stderr parsl.dataflow.dflow INFO: Task 170 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3bc1aaaf48a4f010aeeca3ce86871c38 with result from task 170 parsl.dataflow.dflow DEBUG: Task 170 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/5a082a25-43a5-4799-90cf-d4d80f9e5c6e_calibrate_855042_39.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/5a082a25-43a5-4799-90cf-d4d80f9e5c6e_calibrate_855042_39.stderr parsl.dataflow.dflow INFO: Task 332 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64bf483b735baf23a1456faa55428608 with result from task 332 parsl.dataflow.dflow DEBUG: Task 332 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/fb05422e-4e85-4dfc-8e66-3fe74529f27d_calibrate_855043_62.stdout parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/fb05422e-4e85-4dfc-8e66-3fe74529f27d_calibrate_855043_62.stderr parsl.dataflow.dflow INFO: Task 989 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 373315e3e94c15ad3788fc73ef948c70 with result from task 989 parsl.dataflow.dflow DEBUG: Task 989 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/c1979fb4-f681-4221-9fdb-ab08ca5412e2_calibrate_855136_51.stdout parsl.dataflow.dflow INFO: Standard error for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/c1979fb4-f681-4221-9fdb-ab08ca5412e2_calibrate_855136_51.stderr parsl.dataflow.dflow INFO: Task 295 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 167de42efc8445e5627c38195d10dbd1 with result from task 295 parsl.dataflow.dflow DEBUG: Task 295 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 296 has memoization hash 3f37f92dddd66500623e8a1f5b1fedbc parsl.dataflow.memoization INFO: Task 296 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 296 try 0 launched on executor multi with executor id 1061 parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/92154ecf-e27c-412f-b4e7-90d757ad1900_calibrate_855138_45.stdout parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/92154ecf-e27c-412f-b4e7-90d757ad1900_calibrate_855138_45.stderr parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/b9df9888-4af4-4cfe-b967-a8c259d0e357_characterizeImage_855138_45.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/b9df9888-4af4-4cfe-b967-a8c259d0e357_characterizeImage_855138_45.stderr parsl.dataflow.dflow INFO: Task 1067 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef0676ff8fa0f13ed20bf5368a82c168 with result from task 1067 parsl.dataflow.dflow DEBUG: Task 1067 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/ab1b1548-4c9d-41d6-b7bf-bae8cc61117e_calibrate_855138_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/ab1b1548-4c9d-41d6-b7bf-bae8cc61117e_calibrate_855138_19.stderr parsl.dataflow.dflow INFO: Task 341 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd763083c68b271396d0ba3da5e8a641 with result from task 341 parsl.dataflow.dflow DEBUG: Task 341 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/eeb90391-ad6e-4121-b8ae-9efab92cea3b_calibrate_855137_47.stdout parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/eeb90391-ad6e-4121-b8ae-9efab92cea3b_calibrate_855137_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:09 proc_lsst.multi:146 [INFO] found job 16912361 in provider checkpoint proc_lsst.multi INFO: found job 16912361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 494 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0b5adfc0a20db257afa3d2d457c6fa2 with result from task 494 parsl.dataflow.dflow DEBUG: Task 494 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/f5880d76-1cbb-4b7e-a3df-856a4464ca1c_calibrate_855043_3.stdout parsl.dataflow.dflow INFO: Standard error for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/f5880d76-1cbb-4b7e-a3df-856a4464ca1c_calibrate_855043_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:09 proc_lsst.multi:146 [INFO] found job 16912362 in provider checkpoint proc_lsst.multi INFO: found job 16912362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 811 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f212957eeafa60cd4fe2b83477efb1c with result from task 811 parsl.dataflow.dflow DEBUG: Task 811 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 812 has memoization hash 70a89e5206d73f5f485b33ca6705fa3a parsl.dataflow.memoization INFO: Task 812 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4560f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 812 try 0 launched on executor multi with executor id 1062 parsl.dataflow.dflow INFO: Standard output for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/f19fcdcb-73f7-4e9c-8049-530dda9e1beb_calibrate_855044_36.stdout parsl.dataflow.dflow INFO: Standard error for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/f19fcdcb-73f7-4e9c-8049-530dda9e1beb_calibrate_855044_36.stderr parsl.dataflow.dflow INFO: Standard output for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/bd65cc3a-afd4-4aca-983d-1ccd5e61622e_characterizeImage_855044_36.stdout parsl.dataflow.dflow INFO: Standard error for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/bd65cc3a-afd4-4aca-983d-1ccd5e61622e_characterizeImage_855044_36.stderr parsl.dataflow.dflow INFO: Task 299 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2ade6c1732f8967c5b7bef827d0d355 with result from task 299 parsl.dataflow.dflow DEBUG: Task 299 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/c99ecf73-e117-476e-941d-cd954cd7a905_calibrate_855137_43.stdout parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/c99ecf73-e117-476e-941d-cd954cd7a905_calibrate_855137_43.stderr parsl.dataflow.dflow INFO: Task 275 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd5e3e53535988e6c0141c3f7a5b5d26 with result from task 275 parsl.dataflow.dflow DEBUG: Task 275 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/31deb485-836c-4481-8900-4b3571b6eb8e_calibrate_855136_26.stdout parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/31deb485-836c-4481-8900-4b3571b6eb8e_calibrate_855136_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:10 proc_lsst.multi:146 [INFO] found job 16912363 in provider checkpoint proc_lsst.multi INFO: found job 16912363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 1021 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64cf7749d327fa5b317a0848009982d3 with result from task 1021 parsl.dataflow.dflow DEBUG: Task 1021 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1022 has memoization hash 59922ca946b1ab4f3647d334433b9a2f parsl.dataflow.memoization INFO: Task 1022 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45540f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1022 try 0 launched on executor multi with executor id 1063 parsl.dataflow.dflow INFO: Standard output for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/1ea97871-09e5-4239-afd8-6e1662a19500_calibrate_855042_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/1ea97871-09e5-4239-afd8-6e1662a19500_calibrate_855042_60.stderr parsl.dataflow.dflow INFO: Standard output for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/e856011a-bbee-491b-b894-ea7c7f837ce6_characterizeImage_855042_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/e856011a-bbee-491b-b894-ea7c7f837ce6_characterizeImage_855042_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:10 proc_lsst.multi:146 [INFO] found job 16912364 in provider checkpoint proc_lsst.multi INFO: found job 16912364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 1049 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f24e9ffe243adede1f632750e76b496b with result from task 1049 parsl.dataflow.dflow DEBUG: Task 1049 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/21c4298a-766b-418a-896f-33e54fe89c84_calibrate_855137_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/21c4298a-766b-418a-896f-33e54fe89c84_calibrate_855137_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:11 proc_lsst.multi:146 [INFO] found job 16912365 in provider checkpoint proc_lsst.multi INFO: found job 16912365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 1084 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa1c4171a4213ec13e0cdfaf92bb1a77 with result from task 1084 parsl.dataflow.dflow DEBUG: Task 1084 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1085 has memoization hash 34626c122d30bd3b64a4de59b1cfcfcc parsl.dataflow.memoization INFO: Task 1085 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45390680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1085 try 0 launched on executor multi with executor id 1064 parsl.dataflow.dflow INFO: Standard output for task 1085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/371dca44-9cbe-4f7c-a0c7-67664229bb57_calibrate_855136_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/371dca44-9cbe-4f7c-a0c7-67664229bb57_calibrate_855136_19.stderr parsl.dataflow.dflow INFO: Standard output for task 1084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/767089d1-fccb-463e-ac23-0f7b17b91689_characterizeImage_855136_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/767089d1-fccb-463e-ac23-0f7b17b91689_characterizeImage_855136_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:11 proc_lsst.multi:146 [INFO] found job 16912366 in provider checkpoint proc_lsst.multi INFO: found job 16912366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:11 proc_lsst.multi:146 [INFO] found job 16912367 in provider checkpoint proc_lsst.multi INFO: found job 16912367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:11 proc_lsst.multi:146 [INFO] found job 16912368 in provider checkpoint proc_lsst.multi INFO: found job 16912368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 316 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df081dee189cab85646a741444afb804 with result from task 316 parsl.dataflow.dflow DEBUG: Task 316 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 317 has memoization hash 6746aa9e3841bda4f67466702f2b3438 parsl.dataflow.memoization INFO: Task 317 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 317 try 0 launched on executor multi with executor id 1065 parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/c675cc9a-7dd0-4660-b640-18b3fe7b9801_calibrate_855043_26.stdout parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/c675cc9a-7dd0-4660-b640-18b3fe7b9801_calibrate_855043_26.stderr parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/25b31e6d-ad58-411a-a7a2-ecbb1d0aa186_characterizeImage_855043_26.stdout parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/25b31e6d-ad58-411a-a7a2-ecbb1d0aa186_characterizeImage_855043_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:12 proc_lsst.multi:146 [INFO] found job 16912369 in provider checkpoint proc_lsst.multi INFO: found job 16912369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:12 proc_lsst.multi:146 [INFO] found job 16912370 in provider checkpoint proc_lsst.multi INFO: found job 16912370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:12 proc_lsst.multi:146 [INFO] found job 16912371 in provider checkpoint proc_lsst.multi INFO: found job 16912371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 734 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e08de78c367d8f798e95b81f0b9353ba with result from task 734 parsl.dataflow.dflow DEBUG: Task 734 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/8030e4c1-4790-422c-a7cd-177aab6b8d44_calibrate_855136_29.stdout parsl.dataflow.dflow INFO: Standard error for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/8030e4c1-4790-422c-a7cd-177aab6b8d44_calibrate_855136_29.stderr parsl.dataflow.dflow INFO: Task 512 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 043cef6d221aed96de7b3d78f012b354 with result from task 512 parsl.dataflow.dflow DEBUG: Task 512 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/977fe3a5-819f-4f77-a8c0-de5809e5055a_calibrate_855042_44.stdout parsl.dataflow.dflow INFO: Standard error for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/977fe3a5-819f-4f77-a8c0-de5809e5055a_calibrate_855042_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:12 proc_lsst.multi:146 [INFO] found job 16912372 in provider checkpoint proc_lsst.multi INFO: found job 16912372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 547 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4bbc6e377e9b0a7069c6498046237c8c with result from task 547 parsl.dataflow.dflow DEBUG: Task 547 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 548 has memoization hash 71cbeaa47a276a24b194051167c37b89 parsl.dataflow.memoization INFO: Task 548 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45701940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 548 try 0 launched on executor multi with executor id 1066 parsl.dataflow.dflow INFO: Standard output for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/d4aa6e07-75c9-4a3d-9bd2-63268d85c214_calibrate_855138_62.stdout parsl.dataflow.dflow INFO: Standard error for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/d4aa6e07-75c9-4a3d-9bd2-63268d85c214_calibrate_855138_62.stderr parsl.dataflow.dflow INFO: Standard output for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/4315ea40-e446-4506-b992-805adc806852_characterizeImage_855138_62.stdout parsl.dataflow.dflow INFO: Standard error for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/4315ea40-e446-4506-b992-805adc806852_characterizeImage_855138_62.stderr parsl.dataflow.dflow INFO: Task 449 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab198a0b102df71c31521c79e004bcbd with result from task 449 parsl.dataflow.dflow DEBUG: Task 449 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/1ba9cadd-64dc-4c20-96e4-4f2f8b0d2300_calibrate_855137_49.stdout parsl.dataflow.dflow INFO: Standard error for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/1ba9cadd-64dc-4c20-96e4-4f2f8b0d2300_calibrate_855137_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:13 proc_lsst.multi:146 [INFO] found job 16912373 in provider checkpoint proc_lsst.multi INFO: found job 16912373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 650 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ae6180eefdda8c02843bb86321c55b3 with result from task 650 parsl.dataflow.dflow DEBUG: Task 650 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/b6e2b540-68e4-4956-a685-fae8fb13fa12_calibrate_855137_26.stdout parsl.dataflow.dflow INFO: Standard error for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/b6e2b540-68e4-4956-a685-fae8fb13fa12_calibrate_855137_26.stderr parsl.dataflow.dflow INFO: Task 400 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 714b50fd16878b26130714fbb823899b with result from task 400 parsl.dataflow.dflow DEBUG: Task 400 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 401 has memoization hash ed26126cd68206c1bb2c92ea6fca44e1 parsl.dataflow.memoization INFO: Task 401 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a7c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 401 try 0 launched on executor multi with executor id 1067 parsl.dataflow.dflow INFO: Standard output for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/4f1275b0-9906-4350-95cf-d3f2b0dc8faf_calibrate_855137_27.stdout parsl.dataflow.dflow INFO: Standard error for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/4f1275b0-9906-4350-95cf-d3f2b0dc8faf_calibrate_855137_27.stderr parsl.dataflow.dflow INFO: Standard output for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/5a8c3a3a-48ee-4092-ab06-9b0527d7602c_characterizeImage_855137_27.stdout parsl.dataflow.dflow INFO: Standard error for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/5a8c3a3a-48ee-4092-ab06-9b0527d7602c_characterizeImage_855137_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:13 proc_lsst.multi:146 [INFO] found job 16912374 in provider checkpoint proc_lsst.multi INFO: found job 16912374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:13 proc_lsst.multi:146 [INFO] found job 16912375 in provider checkpoint proc_lsst.multi INFO: found job 16912375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 937 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1035ec8e36593db0ed35beef72c06d14 with result from task 937 parsl.dataflow.dflow DEBUG: Task 937 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 938 has memoization hash 3c40ca1ff6bd0278846cbf6d737e9a0c parsl.dataflow.memoization INFO: Task 938 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4549b7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 938 try 0 launched on executor multi with executor id 1068 parsl.dataflow.dflow INFO: Standard output for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/a72031b4-6530-40ab-af9a-58beb86e9a64_calibrate_855138_17.stdout parsl.dataflow.dflow INFO: Standard error for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/a72031b4-6530-40ab-af9a-58beb86e9a64_calibrate_855138_17.stderr parsl.dataflow.dflow INFO: Standard output for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/8757df07-08b1-4025-a193-92c2aa241ecd_characterizeImage_855138_17.stdout parsl.dataflow.dflow INFO: Standard error for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/8757df07-08b1-4025-a193-92c2aa241ecd_characterizeImage_855138_17.stderr parsl.dataflow.dflow INFO: Task 851 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e9dc6975329cf2ea4c98cb4a16be16d with result from task 851 parsl.dataflow.dflow DEBUG: Task 851 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/3a849d7d-aa12-4d69-9c13-e8b718809809_calibrate_855042_54.stdout parsl.dataflow.dflow INFO: Standard error for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/3a849d7d-aa12-4d69-9c13-e8b718809809_calibrate_855042_54.stderr parsl.dataflow.dflow INFO: Task 544 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c30519cbd1108bdea301a66c6c0119eb with result from task 544 parsl.dataflow.dflow DEBUG: Task 544 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 545 has memoization hash ebd574f2d1213096292dc4b33d573e65 parsl.dataflow.memoization INFO: Task 545 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45701620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 545 try 0 launched on executor multi with executor id 1069 parsl.dataflow.dflow INFO: Standard output for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/ec6dbe93-0eb2-49d0-9ac2-dee432d8571f_calibrate_855138_18.stdout parsl.dataflow.dflow INFO: Standard error for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/ec6dbe93-0eb2-49d0-9ac2-dee432d8571f_calibrate_855138_18.stderr parsl.dataflow.dflow INFO: Standard output for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/62d5cdfa-e4a5-4dc8-affc-7b7288eb1429_characterizeImage_855138_18.stdout parsl.dataflow.dflow INFO: Standard error for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/62d5cdfa-e4a5-4dc8-affc-7b7288eb1429_characterizeImage_855138_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:14 proc_lsst.multi:146 [INFO] found job 16912378 in provider checkpoint proc_lsst.multi INFO: found job 16912378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 1034 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e008a7936b9cb992940cb608f3b1f66 with result from task 1034 parsl.dataflow.dflow DEBUG: Task 1034 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/3439bc5e-a29d-4a65-ba28-6f3fe022e3d2_calibrate_855043_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/3439bc5e-a29d-4a65-ba28-6f3fe022e3d2_calibrate_855043_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 137 parsl.dataflow.strategy DEBUG: Executor multi has 137 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 2024-03-08 07:20:14 proc_lsst.multi:146 [INFO] found job 75775 in provider local proc_lsst.multi INFO: found job 75775 in provider local 2024-03-08 07:20:14 proc_lsst.multi:146 [INFO] found job 16912283 in provider astro proc_lsst.multi INFO: found job 16912283 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:14 proc_lsst.multi:146 [INFO] found job 16912284 in provider astro proc_lsst.multi INFO: found job 16912284 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:14 proc_lsst.multi:146 [INFO] found job 16912285 in provider astro proc_lsst.multi INFO: found job 16912285 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 506 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 969e76bf2ac609617ab6c64472869dff with result from task 506 parsl.dataflow.dflow DEBUG: Task 506 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/163b3d5c-e9d6-44cf-886c-d10f11c2253a_calibrate_855042_15.stdout parsl.dataflow.dflow INFO: Standard error for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/163b3d5c-e9d6-44cf-886c-d10f11c2253a_calibrate_855042_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:15 proc_lsst.multi:146 [INFO] found job 16912286 in provider astro proc_lsst.multi INFO: found job 16912286 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 31 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c8779edab48d420ae9e5fe4b10be53a with result from task 31 parsl.dataflow.dflow DEBUG: Task 31 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 32 has memoization hash ebd16fa14adda3a2f820451ce4a934f8 parsl.dataflow.memoization INFO: Task 32 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467d0180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 32 try 0 launched on executor multi with executor id 1070 parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/9da78cfc-8d0d-4e07-a406-00f3dc8c69ce_calibrate_855043_61.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/9da78cfc-8d0d-4e07-a406-00f3dc8c69ce_calibrate_855043_61.stderr parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/ce323cc8-de0d-4390-996f-f4fe562d5971_characterizeImage_855043_61.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/ce323cc8-de0d-4390-996f-f4fe562d5971_characterizeImage_855043_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:15 proc_lsst.multi:146 [INFO] found job 16912287 in provider astro proc_lsst.multi INFO: found job 16912287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 1024 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f18dc60954ca83dafa2dc533ee0e136 with result from task 1024 parsl.dataflow.dflow DEBUG: Task 1024 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1025 has memoization hash e808fefd4951e35b44512d64e802d2c5 parsl.dataflow.memoization INFO: Task 1025 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45541260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1025 try 0 launched on executor multi with executor id 1071 parsl.dataflow.dflow INFO: Standard output for task 1025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/92099d65-d254-443e-a8dc-3ab48565e3e0_calibrate_855043_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/92099d65-d254-443e-a8dc-3ab48565e3e0_calibrate_855043_11.stderr parsl.dataflow.dflow INFO: Standard output for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/5882e7dd-b851-42c7-87c2-0deaa58ff76f_characterizeImage_855043_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/5882e7dd-b851-42c7-87c2-0deaa58ff76f_characterizeImage_855043_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:15 proc_lsst.multi:146 [INFO] found job 16912288 in provider astro proc_lsst.multi INFO: found job 16912288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 542 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9a73d5b0cf723e8fc6cf5507b18c942 with result from task 542 parsl.dataflow.dflow DEBUG: Task 542 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/26899f1f-ba0f-4565-ba4d-44eaa1e1a54d_calibrate_855044_20.stdout parsl.dataflow.dflow INFO: Standard error for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/26899f1f-ba0f-4565-ba4d-44eaa1e1a54d_calibrate_855044_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:15 proc_lsst.multi:146 [INFO] found job 16912289 in provider astro proc_lsst.multi INFO: found job 16912289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 478 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7dd3bbc8d7a07566c0f7615d9bcc70fc with result from task 478 parsl.dataflow.dflow DEBUG: Task 478 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 479 has memoization hash f9c414a40c5ec349a3435badd8180c8f parsl.dataflow.memoization INFO: Task 479 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456cca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 479 try 0 launched on executor multi with executor id 1072 parsl.dataflow.dflow INFO: Standard output for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/fcdc7998-0a4f-4f44-a083-fcd9b47b2296_calibrate_855043_60.stdout parsl.dataflow.dflow INFO: Standard error for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/fcdc7998-0a4f-4f44-a083-fcd9b47b2296_calibrate_855043_60.stderr parsl.dataflow.dflow INFO: Standard output for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/be51a605-79de-4a6a-8ea5-f93ec2f4e381_characterizeImage_855043_60.stdout parsl.dataflow.dflow INFO: Standard error for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/be51a605-79de-4a6a-8ea5-f93ec2f4e381_characterizeImage_855043_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:16 proc_lsst.multi:146 [INFO] found job 16912290 in provider astro proc_lsst.multi INFO: found job 16912290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 349 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb0869a284e0f7ac37722ec2979bcf58 with result from task 349 parsl.dataflow.dflow DEBUG: Task 349 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 350 has memoization hash f54ede638daba2636bb73c80bd523bcf parsl.dataflow.memoization INFO: Task 350 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46878f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 350 try 0 launched on executor multi with executor id 1073 parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/ed68ce22-c954-4f9b-9ef0-2cd474c58f49_calibrate_855136_30.stdout parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/ed68ce22-c954-4f9b-9ef0-2cd474c58f49_calibrate_855136_30.stderr parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/e2940e99-8666-4245-9f6d-fdd7edb6a5df_characterizeImage_855136_30.stdout parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/e2940e99-8666-4245-9f6d-fdd7edb6a5df_characterizeImage_855136_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:16 proc_lsst.multi:146 [INFO] found job 16912291 in provider astro proc_lsst.multi INFO: found job 16912291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:16 proc_lsst.multi:146 [INFO] found job 16912292 in provider astro proc_lsst.multi INFO: found job 16912292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 947 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d25ab06265ed52398ff20e67f2a01865 with result from task 947 parsl.dataflow.dflow DEBUG: Task 947 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/d9821108-412f-4b6d-b861-ec692152f986_calibrate_855136_22.stdout parsl.dataflow.dflow INFO: Standard error for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/d9821108-412f-4b6d-b861-ec692152f986_calibrate_855136_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:17 proc_lsst.multi:146 [INFO] found job 16912293 in provider astro proc_lsst.multi INFO: found job 16912293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 635 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1af496d4449423c53b4c0c34b93697be with result from task 635 parsl.dataflow.dflow DEBUG: Task 635 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/9fb62733-f011-4bb0-a94d-301139db1405_calibrate_855044_37.stdout parsl.dataflow.dflow INFO: Standard error for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/9fb62733-f011-4bb0-a94d-301139db1405_calibrate_855044_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:17 proc_lsst.multi:146 [INFO] found job 16912294 in provider astro proc_lsst.multi INFO: found job 16912294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 622 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ddf7f7138d9aaf0e49096a5cfe8f4ca with result from task 622 parsl.dataflow.dflow DEBUG: Task 622 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 623 has memoization hash 27be416df4ec0d828f39ed583412d9e9 parsl.dataflow.memoization INFO: Task 623 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa457422a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 623 try 0 launched on executor multi with executor id 1074 parsl.dataflow.dflow INFO: Standard output for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/31cc046b-9a88-4ba3-93da-e960a5114781_calibrate_855136_12.stdout parsl.dataflow.dflow INFO: Standard error for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/31cc046b-9a88-4ba3-93da-e960a5114781_calibrate_855136_12.stderr parsl.dataflow.dflow INFO: Standard output for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/b97e250c-4e6a-4f37-86d7-a4a7c4f9ac6c_characterizeImage_855136_12.stdout parsl.dataflow.dflow INFO: Standard error for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/b97e250c-4e6a-4f37-86d7-a4a7c4f9ac6c_characterizeImage_855136_12.stderr parsl.dataflow.dflow INFO: Task 560 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10e7485419ed6a7e7d019bcbde1e7b6e with result from task 560 parsl.dataflow.dflow DEBUG: Task 560 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/289d1aac-915b-44a3-807a-47b8a9c66ee5_calibrate_855138_50.stdout parsl.dataflow.dflow INFO: Standard error for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/289d1aac-915b-44a3-807a-47b8a9c66ee5_calibrate_855138_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:17 proc_lsst.multi:146 [INFO] found job 16912295 in provider astro proc_lsst.multi INFO: found job 16912295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 1001 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b505e34c10e773072bfe4c0379e1f995 with result from task 1001 parsl.dataflow.dflow DEBUG: Task 1001 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/b86ea807-bd1e-4af6-83b8-f35d441405bd_calibrate_855044_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/b86ea807-bd1e-4af6-83b8-f35d441405bd_calibrate_855044_49.stderr parsl.dataflow.dflow INFO: Task 191 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 34fc7ef56ed9875808b286d2c363d106 with result from task 191 parsl.dataflow.dflow DEBUG: Task 191 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/91f7512b-941c-4cba-81ed-498e70d0ff7e_calibrate_855044_60.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/91f7512b-941c-4cba-81ed-498e70d0ff7e_calibrate_855044_60.stderr parsl.dataflow.dflow INFO: Task 1075 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7037946e2b120df4fc2aca0f9388d9d0 with result from task 1075 parsl.dataflow.dflow DEBUG: Task 1075 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1076 has memoization hash 707a8e11c554b042364e2bc74d99deba parsl.dataflow.memoization INFO: Task 1076 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45392ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1076 try 0 launched on executor multi with executor id 1075 parsl.dataflow.dflow INFO: Standard output for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/3248ee6a-5a81-4c7b-ab13-053e91f22e4e_calibrate_855136_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/3248ee6a-5a81-4c7b-ab13-053e91f22e4e_calibrate_855136_46.stderr parsl.dataflow.dflow INFO: Standard output for task 1075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/f689e8c5-cfdf-42d7-b014-22d0462efd76_characterizeImage_855136_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/f689e8c5-cfdf-42d7-b014-22d0462efd76_characterizeImage_855136_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:17 proc_lsst.multi:146 [INFO] found job 16912296 in provider astro proc_lsst.multi INFO: found job 16912296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 1018 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1005e0d1f908c1516755ed5e2d8b799a with result from task 1018 parsl.dataflow.dflow DEBUG: Task 1018 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1019 has memoization hash 05f9dc1a4d3a653ca851657afd46d72e parsl.dataflow.memoization INFO: Task 1019 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455409a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1019 try 0 launched on executor multi with executor id 1076 parsl.dataflow.dflow INFO: Standard output for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/a6b7afb7-9a6d-40fe-82ca-6084bc5aa64d_calibrate_855137_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/a6b7afb7-9a6d-40fe-82ca-6084bc5aa64d_calibrate_855137_24.stderr parsl.dataflow.dflow INFO: Standard output for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/8dbb4b2f-d5bb-4f3f-a084-60111c0464a2_characterizeImage_855137_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/8dbb4b2f-d5bb-4f3f-a084-60111c0464a2_characterizeImage_855137_24.stderr parsl.dataflow.dflow INFO: Task 929 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4fc6ddcde3a65955fbd1fc3750adfcc with result from task 929 parsl.dataflow.dflow DEBUG: Task 929 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/1568681b-39f8-48d7-adc8-183389975fbd_calibrate_855138_25.stdout parsl.dataflow.dflow INFO: Standard error for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/1568681b-39f8-48d7-adc8-183389975fbd_calibrate_855138_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:18 proc_lsst.multi:146 [INFO] found job 16912297 in provider astro proc_lsst.multi INFO: found job 16912297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:18 proc_lsst.multi:146 [INFO] found job 16912298 in provider astro proc_lsst.multi INFO: found job 16912298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 440 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f96b292e90451f8e11b1a0a2b56ee9ef with result from task 440 parsl.dataflow.dflow DEBUG: Task 440 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/301130dd-5229-4eaa-9666-c21f9ce7ad3c_calibrate_855044_23.stdout parsl.dataflow.dflow INFO: Standard error for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/301130dd-5229-4eaa-9666-c21f9ce7ad3c_calibrate_855044_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:18 proc_lsst.multi:146 [INFO] found job 16912299 in provider astro proc_lsst.multi INFO: found job 16912299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:18 proc_lsst.multi:146 [INFO] found job 16912300 in provider astro proc_lsst.multi INFO: found job 16912300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 538 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 466e02d665a5de1c81b29ba06a307cd5 with result from task 538 parsl.dataflow.dflow DEBUG: Task 538 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 539 has memoization hash 7bf459a1ff0061cbbffadc785e8b70b4 parsl.dataflow.memoization INFO: Task 539 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45700fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 539 try 0 launched on executor multi with executor id 1077 parsl.dataflow.dflow INFO: Standard output for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/cfe59608-3d54-46b4-9d04-dd622d46cd15_calibrate_855043_10.stdout parsl.dataflow.dflow INFO: Standard error for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/cfe59608-3d54-46b4-9d04-dd622d46cd15_calibrate_855043_10.stderr parsl.dataflow.dflow INFO: Standard output for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/9376f528-6e14-4f1a-9383-16a250ef90a3_characterizeImage_855043_10.stdout parsl.dataflow.dflow INFO: Standard error for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/9376f528-6e14-4f1a-9383-16a250ef90a3_characterizeImage_855043_10.stderr parsl.dataflow.dflow INFO: Task 670 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ec92eb4f257598a16e6a452a4d5456c with result from task 670 parsl.dataflow.dflow DEBUG: Task 670 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 671 has memoization hash b3cafef8eb3e375d3bf9a023ea75c56e parsl.dataflow.memoization INFO: Task 671 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 671 try 0 launched on executor multi with executor id 1078 parsl.dataflow.dflow INFO: Standard output for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/1fe8ddc2-d9e5-4ba4-b064-8bcd3a55c1dd_calibrate_855137_55.stdout parsl.dataflow.dflow INFO: Standard error for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/1fe8ddc2-d9e5-4ba4-b064-8bcd3a55c1dd_calibrate_855137_55.stderr parsl.dataflow.dflow INFO: Standard output for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/e8a1316a-9416-4158-bd33-ea6782e35678_characterizeImage_855137_55.stdout parsl.dataflow.dflow INFO: Standard error for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/e8a1316a-9416-4158-bd33-ea6782e35678_characterizeImage_855137_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:19 proc_lsst.multi:146 [INFO] found job 16912301 in provider astro proc_lsst.multi INFO: found job 16912301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 155 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 812f07c4ebb8a1181c3ea1148a470350 with result from task 155 parsl.dataflow.dflow DEBUG: Task 155 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/e45ff0a2-1822-444a-b540-12e6ed46e5a4_calibrate_855138_56.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/e45ff0a2-1822-444a-b540-12e6ed46e5a4_calibrate_855138_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:19 proc_lsst.multi:146 [INFO] found job 16912302 in provider astro proc_lsst.multi INFO: found job 16912302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:19 proc_lsst.multi:146 [INFO] found job 16912303 in provider astro proc_lsst.multi INFO: found job 16912303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:20 proc_lsst.multi:146 [INFO] found job 16912304 in provider astro proc_lsst.multi INFO: found job 16912304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 1078 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b6e2433c77050931dfa29a14e897c438 with result from task 1078 parsl.dataflow.dflow DEBUG: Task 1078 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1079 has memoization hash 1c8fc0b31750f4a6a6ce387e843d9b4c parsl.dataflow.memoization INFO: Task 1079 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45390360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1079 try 0 launched on executor multi with executor id 1079 parsl.dataflow.dflow INFO: Standard output for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/24eda5b8-551c-41fe-8df2-adc2f078a02d_calibrate_855138_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/24eda5b8-551c-41fe-8df2-adc2f078a02d_calibrate_855138_28.stderr parsl.dataflow.dflow INFO: Standard output for task 1078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/9d1f5f35-5552-464f-b077-593393c3f77a_characterizeImage_855138_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/9d1f5f35-5552-464f-b077-593393c3f77a_characterizeImage_855138_28.stderr parsl.dataflow.dflow INFO: Task 1072 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1662213a7df9f9b7197b055e58d39580 with result from task 1072 parsl.dataflow.dflow DEBUG: Task 1072 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1073 has memoization hash 29665f245ace2d43998ec6c450f7cbf4 parsl.dataflow.memoization INFO: Task 1073 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45392de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1073 try 0 launched on executor multi with executor id 1080 parsl.dataflow.dflow INFO: Standard output for task 1073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/83156d7e-3484-479c-8f74-ac6ded3ccae5_calibrate_855136_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/83156d7e-3484-479c-8f74-ac6ded3ccae5_calibrate_855136_24.stderr parsl.dataflow.dflow INFO: Standard output for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/0275787f-ea3a-4b32-909a-45e4bbbcfd6e_characterizeImage_855136_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/0275787f-ea3a-4b32-909a-45e4bbbcfd6e_characterizeImage_855136_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:20 proc_lsst.multi:146 [INFO] found job 16912305 in provider astro proc_lsst.multi INFO: found job 16912305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 673 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dca7fab4b28fa9b9ce6c00ebbab33c97 with result from task 673 parsl.dataflow.dflow DEBUG: Task 673 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 674 has memoization hash 55120e13ffe4f66da32ead03db6ef320 parsl.dataflow.memoization INFO: Task 674 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 674 try 0 launched on executor multi with executor id 1081 parsl.dataflow.dflow INFO: Standard output for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/025748f1-2ac5-41a5-bc12-8af38ef13695_calibrate_855043_16.stdout parsl.dataflow.dflow INFO: Standard error for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/025748f1-2ac5-41a5-bc12-8af38ef13695_calibrate_855043_16.stderr parsl.dataflow.dflow INFO: Standard output for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/2c676aad-45b1-47a2-833e-2e0eb389ad63_characterizeImage_855043_16.stdout parsl.dataflow.dflow INFO: Standard error for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/2c676aad-45b1-47a2-833e-2e0eb389ad63_characterizeImage_855043_16.stderr parsl.dataflow.dflow INFO: Task 1093 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32fb35da4f500e649101280e028f260c with result from task 1093 parsl.dataflow.dflow DEBUG: Task 1093 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1094 has memoization hash 245982a3860944a9c31d20ad28df53e7 parsl.dataflow.memoization INFO: Task 1094 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45391120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1094 try 0 launched on executor multi with executor id 1082 parsl.dataflow.dflow INFO: Standard output for task 1094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/9624e48a-6724-4045-867f-0e20d9956ebd_calibrate_855043_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/9624e48a-6724-4045-867f-0e20d9956ebd_calibrate_855043_23.stderr parsl.dataflow.dflow INFO: Standard output for task 1093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/a2168549-6764-437e-9864-0b7049889c86_characterizeImage_855043_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855043/a2168549-6764-437e-9864-0b7049889c86_characterizeImage_855043_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:20 proc_lsst.multi:146 [INFO] found job 16912306 in provider astro proc_lsst.multi INFO: found job 16912306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 1042 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 412793f7a6f58799ba29a6c1be86a6a3 with result from task 1042 parsl.dataflow.dflow DEBUG: Task 1042 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1043 has memoization hash 5f1ba72f28515fad94cf11cb1d7d248f parsl.dataflow.memoization INFO: Task 1043 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45542520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1043 try 0 launched on executor multi with executor id 1083 parsl.dataflow.dflow INFO: Standard output for task 1043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/4434f278-e59d-4ce6-bdd7-12ca4d2c8aa2_calibrate_855136_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/4434f278-e59d-4ce6-bdd7-12ca4d2c8aa2_calibrate_855136_16.stderr parsl.dataflow.dflow INFO: Standard output for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/34e5a714-87e6-4c51-8cc5-5b139042af69_characterizeImage_855136_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/34e5a714-87e6-4c51-8cc5-5b139042af69_characterizeImage_855136_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:21 proc_lsst.multi:146 [INFO] found job 16912307 in provider astro proc_lsst.multi INFO: found job 16912307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:21 proc_lsst.multi:146 [INFO] found job 16912308 in provider astro proc_lsst.multi INFO: found job 16912308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:21 proc_lsst.multi:146 [INFO] found job 16912309 in provider astro proc_lsst.multi INFO: found job 16912309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 599 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf6558b039db453d16036087a5b4e241 with result from task 599 parsl.dataflow.dflow DEBUG: Task 599 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/f71abd10-fca0-40ea-87f5-26a475dd79d6_calibrate_855042_33.stdout parsl.dataflow.dflow INFO: Standard error for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/f71abd10-fca0-40ea-87f5-26a475dd79d6_calibrate_855042_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:21 proc_lsst.multi:146 [INFO] found job 16912310 in provider astro proc_lsst.multi INFO: found job 16912310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 496 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ccd050e6609047e3469b6aab6946d982 with result from task 496 parsl.dataflow.dflow DEBUG: Task 496 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 497 has memoization hash 4b43c89683ebf8a25ac41787aa33c56c parsl.dataflow.memoization INFO: Task 497 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456cdd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 497 try 0 launched on executor multi with executor id 1084 parsl.dataflow.dflow INFO: Standard output for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/9ba36f31-072d-44c5-ab51-27cf2ba38957_calibrate_855138_51.stdout parsl.dataflow.dflow INFO: Standard error for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/9ba36f31-072d-44c5-ab51-27cf2ba38957_calibrate_855138_51.stderr parsl.dataflow.dflow INFO: Standard output for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/da0628c6-8ff5-4687-90d5-129e09a2251b_characterizeImage_855138_51.stdout parsl.dataflow.dflow INFO: Standard error for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/da0628c6-8ff5-4687-90d5-129e09a2251b_characterizeImage_855138_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:22 proc_lsst.multi:146 [INFO] found job 16912311 in provider astro proc_lsst.multi INFO: found job 16912311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:22 proc_lsst.multi:146 [INFO] found job 16912312 in provider astro proc_lsst.multi INFO: found job 16912312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 782 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d57289390e0dc16e80814c56fa5c34c0 with result from task 782 parsl.dataflow.dflow DEBUG: Task 782 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/2dd59477-04b8-477c-b030-24598cbfdfbb_calibrate_855042_22.stdout parsl.dataflow.dflow INFO: Standard error for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/2dd59477-04b8-477c-b030-24598cbfdfbb_calibrate_855042_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:22 proc_lsst.multi:146 [INFO] found job 16912313 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912313 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.dataflow.dflow INFO: Task 985 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 429b164cda52711389ced1c86627f3a7 with result from task 985 parsl.dataflow.dflow DEBUG: Task 985 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 986 has memoization hash 3fc07bdc851fd8472ee2779a727380fc parsl.dataflow.memoization INFO: Task 986 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454f9b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 986 try 0 launched on executor multi with executor id 1085 parsl.dataflow.dflow INFO: Standard output for task 986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/b7329003-0c45-40c6-a1a6-5aed84f3269d_calibrate_855138_20.stdout parsl.dataflow.dflow INFO: Standard error for task 986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/b7329003-0c45-40c6-a1a6-5aed84f3269d_calibrate_855138_20.stderr parsl.dataflow.dflow INFO: Standard output for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/9db6b781-cf39-456a-974f-f9f6baa137fa_characterizeImage_855138_20.stdout parsl.dataflow.dflow INFO: Standard error for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/9db6b781-cf39-456a-974f-f9f6baa137fa_characterizeImage_855138_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:22 proc_lsst.multi:146 [INFO] found job 16912314 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912314 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.dataflow.dflow INFO: Task 80 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5b00c3ff7f27688d2f6f6a659a332a1 with result from task 80 parsl.dataflow.dflow DEBUG: Task 80 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/0fdc6710-c726-464f-bcd3-bbd2bd4df789_calibrate_855136_60.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/0fdc6710-c726-464f-bcd3-bbd2bd4df789_calibrate_855136_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:23 proc_lsst.multi:146 [INFO] found job 16912315 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912315 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:23 proc_lsst.multi:146 [INFO] found job 16912316 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912316 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:23 proc_lsst.multi:146 [INFO] found job 16912317 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:24 proc_lsst.multi:146 [INFO] found job 16912318 in provider escience_a40 proc_lsst.multi INFO: found job 16912318 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:24 proc_lsst.multi:146 [INFO] found job 16912319 in provider escience_a40 proc_lsst.multi INFO: found job 16912319 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 355 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7a74492eb9b91393247b6ff904c89e9 with result from task 355 parsl.dataflow.dflow DEBUG: Task 355 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 356 has memoization hash 700a9a8efe18706da0f71447cb9854aa parsl.dataflow.memoization INFO: Task 356 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46879580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 356 try 0 launched on executor multi with executor id 1086 parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/7caf21bb-73b1-450b-abec-1a3e56063de4_calibrate_855138_10.stdout parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/7caf21bb-73b1-450b-abec-1a3e56063de4_calibrate_855138_10.stderr parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/128dd7f0-3993-430b-a0b4-2241863aadad_characterizeImage_855138_10.stdout parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/128dd7f0-3993-430b-a0b4-2241863aadad_characterizeImage_855138_10.stderr parsl.dataflow.dflow INFO: Task 1070 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 072a03a5183bf0d28047f87d5482e2ce with result from task 1070 parsl.dataflow.dflow DEBUG: Task 1070 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/e3e535c0-44e8-4ded-89dc-3351b7c721a1_calibrate_855138_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/e3e535c0-44e8-4ded-89dc-3351b7c721a1_calibrate_855138_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:24 proc_lsst.multi:146 [INFO] found job 16912320 in provider escience_a40 proc_lsst.multi INFO: found job 16912320 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:24 proc_lsst.multi:146 [INFO] found job 16912321 in provider escience_a40 proc_lsst.multi INFO: found job 16912321 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 956 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fcf1668f9bb4ffa3d724a31eb60853c1 with result from task 956 parsl.dataflow.dflow DEBUG: Task 956 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/19250f4e-6071-4aef-9f1d-c04346d79eaa_calibrate_855138_7.stdout parsl.dataflow.dflow INFO: Standard error for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/19250f4e-6071-4aef-9f1d-c04346d79eaa_calibrate_855138_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:25 proc_lsst.multi:146 [INFO] found job 16912322 in provider escience_a40 proc_lsst.multi INFO: found job 16912322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:25 proc_lsst.multi:146 [INFO] found job 16912323 in provider escience_a40 proc_lsst.multi INFO: found job 16912323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 503 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13f57901fbdd0de46a524b9e60cc62ec with result from task 503 parsl.dataflow.dflow DEBUG: Task 503 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/7fc0adcc-1cca-4bab-8788-cc19e86d2e95_calibrate_855042_13.stdout parsl.dataflow.dflow INFO: Standard error for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/7fc0adcc-1cca-4bab-8788-cc19e86d2e95_calibrate_855042_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:25 proc_lsst.multi:146 [INFO] found job 16912324 in provider escience_a40 proc_lsst.multi INFO: found job 16912324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 1111 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7fb4751f3ebe511ed2e632ada5200e81 with result from task 1111 parsl.dataflow.dflow DEBUG: Task 1111 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1112 has memoization hash 863bec0fb881a5cb38460b4b55594ade parsl.dataflow.memoization INFO: Task 1112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45392340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1112 try 0 launched on executor multi with executor id 1087 parsl.dataflow.dflow INFO: Standard output for task 1112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/52dca954-843c-4429-840b-07f639abf291_calibrate_855138_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/52dca954-843c-4429-840b-07f639abf291_calibrate_855138_54.stderr parsl.dataflow.dflow INFO: Standard output for task 1111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/71ef542c-b845-4301-b3a7-05b05407e96c_characterizeImage_855138_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/71ef542c-b845-4301-b3a7-05b05407e96c_characterizeImage_855138_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:25 proc_lsst.multi:146 [INFO] found job 16912325 in provider escience_a40 proc_lsst.multi INFO: found job 16912325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 943 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ead3f24839242e782a5aa6ce2bf8c1b2 with result from task 943 parsl.dataflow.dflow DEBUG: Task 943 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 944 has memoization hash 09e3ad22cfaa96e09eec63b7aa0962b6 parsl.dataflow.memoization INFO: Task 944 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454fade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 944 try 0 launched on executor multi with executor id 1088 parsl.dataflow.dflow INFO: Standard output for task 944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/948c6828-1762-4a20-862f-5c972183e1e9_calibrate_855138_22.stdout parsl.dataflow.dflow INFO: Standard error for task 944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/948c6828-1762-4a20-862f-5c972183e1e9_calibrate_855138_22.stderr parsl.dataflow.dflow INFO: Standard output for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/b0ee0349-a09f-4681-8d2b-8206dc46d8a0_characterizeImage_855138_22.stdout parsl.dataflow.dflow INFO: Standard error for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/b0ee0349-a09f-4681-8d2b-8206dc46d8a0_characterizeImage_855138_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:26 proc_lsst.multi:146 [INFO] found job 16912326 in provider escience_a40 proc_lsst.multi INFO: found job 16912326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 305 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3475df9502128e88168edb80d5bddd64 with result from task 305 parsl.dataflow.dflow DEBUG: Task 305 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/fda72563-cbab-49a9-9437-9fb49253d6ac_calibrate_855044_39.stdout parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/fda72563-cbab-49a9-9437-9fb49253d6ac_calibrate_855044_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:26 proc_lsst.multi:146 [INFO] found job 16912327 in provider escience_a40 proc_lsst.multi INFO: found job 16912327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 1058 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8bd3ea3eff9cb81bc1db67b4e8efb4d with result from task 1058 parsl.dataflow.dflow DEBUG: Task 1058 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/459592e7-0a36-4b6c-aad1-2372d8ea0bd7_calibrate_855042_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/459592e7-0a36-4b6c-aad1-2372d8ea0bd7_calibrate_855042_37.stderr parsl.dataflow.dflow INFO: Task 353 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 032fef6755238c1c697e50569749f3a3 with result from task 353 parsl.dataflow.dflow DEBUG: Task 353 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/08ad799a-07fa-46ae-b322-64792da32545_calibrate_855042_38.stdout parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/08ad799a-07fa-46ae-b322-64792da32545_calibrate_855042_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:26 proc_lsst.multi:146 [INFO] found job 16912328 in provider escience_a40 proc_lsst.multi INFO: found job 16912328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:26 proc_lsst.multi:146 [INFO] found job 16912329 in provider escience_a40 proc_lsst.multi INFO: found job 16912329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:27 proc_lsst.multi:146 [INFO] found job 16912330 in provider escience_a40 proc_lsst.multi INFO: found job 16912330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 1088 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0d89a0cacd4056b7a5a4fa6cd6684ce with result from task 1088 parsl.dataflow.dflow DEBUG: Task 1088 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/e07dd087-a68f-41e1-b4ff-dbb7cd785fed_calibrate_855136_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/e07dd087-a68f-41e1-b4ff-dbb7cd785fed_calibrate_855136_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:27 proc_lsst.multi:146 [INFO] found job 16912331 in provider escience_a40 proc_lsst.multi INFO: found job 16912331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 980 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 824607e4459e64ce7b387656e8f1b727 with result from task 980 parsl.dataflow.dflow DEBUG: Task 980 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/fb556dc9-a9f2-4c0b-a2cb-c567076d9abd_calibrate_855138_11.stdout parsl.dataflow.dflow INFO: Standard error for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/fb556dc9-a9f2-4c0b-a2cb-c567076d9abd_calibrate_855138_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:27 proc_lsst.multi:146 [INFO] found job 16912332 in provider escience_a40 proc_lsst.multi INFO: found job 16912332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 383 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a50a98f7fdfb6c700977af186f5564d with result from task 383 parsl.dataflow.dflow DEBUG: Task 383 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/109a6ba6-944d-4653-84a9-80eb36e49663_calibrate_855042_59.stdout parsl.dataflow.dflow INFO: Standard error for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/109a6ba6-944d-4653-84a9-80eb36e49663_calibrate_855042_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:28 proc_lsst.multi:146 [INFO] found job 16912333 in provider escience_a40 proc_lsst.multi INFO: found job 16912333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:28 proc_lsst.multi:146 [INFO] found job 16912334 in provider escience_a40 proc_lsst.multi INFO: found job 16912334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:28 proc_lsst.multi:146 [INFO] found job 16912335 in provider escience_a40 proc_lsst.multi INFO: found job 16912335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:28 proc_lsst.multi:146 [INFO] found job 16912336 in provider escience_a40 proc_lsst.multi INFO: found job 16912336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:28 proc_lsst.multi:146 [INFO] found job 16912337 in provider escience_a40 proc_lsst.multi INFO: found job 16912337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:29 proc_lsst.multi:146 [INFO] found job 16912338 in provider checkpoint proc_lsst.multi INFO: found job 16912338 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:29 proc_lsst.multi:146 [INFO] found job 16912339 in provider checkpoint proc_lsst.multi INFO: found job 16912339 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:29 proc_lsst.multi:146 [INFO] found job 16912340 in provider checkpoint proc_lsst.multi INFO: found job 16912340 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:29 proc_lsst.multi:146 [INFO] found job 16912341 in provider checkpoint proc_lsst.multi INFO: found job 16912341 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:30 proc_lsst.multi:146 [INFO] found job 16912342 in provider checkpoint proc_lsst.multi INFO: found job 16912342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 688 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6b45dcf2621d197b005ce99ab306736 with result from task 688 parsl.dataflow.dflow DEBUG: Task 688 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 689 has memoization hash b262dbaba84d36d96fc192a010dcfdba parsl.dataflow.memoization INFO: Task 689 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 689 try 0 launched on executor multi with executor id 1089 parsl.dataflow.dflow INFO: Standard output for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/71a29078-3ea8-4acd-b460-363b30550f24_calibrate_855136_15.stdout parsl.dataflow.dflow INFO: Standard error for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/71a29078-3ea8-4acd-b460-363b30550f24_calibrate_855136_15.stderr parsl.dataflow.dflow INFO: Standard output for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/bb4cb75c-1e13-42f5-bf68-6986a2c18fd4_characterizeImage_855136_15.stdout parsl.dataflow.dflow INFO: Standard error for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/bb4cb75c-1e13-42f5-bf68-6986a2c18fd4_characterizeImage_855136_15.stderr parsl.dataflow.dflow INFO: Task 418 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ce77ddc1e7ee59544c53d10046ab314 with result from task 418 parsl.dataflow.dflow DEBUG: Task 418 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 419 has memoization hash 8e516114d3a6e84d4b93825dcfa7a1d9 parsl.dataflow.memoization INFO: Task 419 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456a4680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 419 try 0 launched on executor multi with executor id 1090 parsl.dataflow.dflow INFO: Standard output for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/a429146b-499d-4e56-9389-a195c74062f3_calibrate_855137_59.stdout parsl.dataflow.dflow INFO: Standard error for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/a429146b-499d-4e56-9389-a195c74062f3_calibrate_855137_59.stderr parsl.dataflow.dflow INFO: Standard output for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/e4777fdb-64ac-4696-8b33-e492962b1623_characterizeImage_855137_59.stdout parsl.dataflow.dflow INFO: Standard error for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/e4777fdb-64ac-4696-8b33-e492962b1623_characterizeImage_855137_59.stderr parsl.dataflow.dflow INFO: Task 458 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 266728b56fa93883b920a09d36037d87 with result from task 458 parsl.dataflow.dflow DEBUG: Task 458 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/14501953-964d-4519-b0dd-e6d139c446ea_calibrate_855136_4.stdout parsl.dataflow.dflow INFO: Standard error for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/14501953-964d-4519-b0dd-e6d139c446ea_calibrate_855136_4.stderr parsl.dataflow.dflow INFO: Task 1012 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 438ed5ddb54c0c740485d7cb5620a9b8 with result from task 1012 parsl.dataflow.dflow DEBUG: Task 1012 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1013 has memoization hash 532e86bcdda6ad9e311ae72d15b17e72 parsl.dataflow.memoization INFO: Task 1013 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45543b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1013 try 0 launched on executor multi with executor id 1091 parsl.dataflow.dflow INFO: Standard output for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/7f24bed8-27c7-4c1d-9581-f4927ebb7cc8_calibrate_855138_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/7f24bed8-27c7-4c1d-9581-f4927ebb7cc8_calibrate_855138_59.stderr parsl.dataflow.dflow INFO: Standard output for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/8c313872-7956-4a64-868f-00269892358f_characterizeImage_855138_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/8c313872-7956-4a64-868f-00269892358f_characterizeImage_855138_59.stderr parsl.dataflow.dflow INFO: Task 526 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b0782db124ca6ea432056505545aa2d with result from task 526 parsl.dataflow.dflow DEBUG: Task 526 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 527 has memoization hash 1114376004fab8ae0ea9fee56b1c2c66 parsl.dataflow.memoization INFO: Task 527 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45700ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 527 try 0 launched on executor multi with executor id 1092 parsl.dataflow.dflow INFO: Standard output for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/bf1de49c-1552-4027-a8fd-b497150318b7_calibrate_855042_26.stdout parsl.dataflow.dflow INFO: Standard error for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/bf1de49c-1552-4027-a8fd-b497150318b7_calibrate_855042_26.stderr parsl.dataflow.dflow INFO: Standard output for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/245d99d9-5f43-4cb3-a15a-ba0771ee70fd_characterizeImage_855042_26.stdout parsl.dataflow.dflow INFO: Standard error for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/245d99d9-5f43-4cb3-a15a-ba0771ee70fd_characterizeImage_855042_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:30 proc_lsst.multi:146 [INFO] found job 16912343 in provider checkpoint proc_lsst.multi INFO: found job 16912343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 619 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 58fd415b8fe3b1e8d74ac2565e4a6e2c with result from task 619 parsl.dataflow.dflow DEBUG: Task 619 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 620 has memoization hash 0593c9e6ac0f131b801ba0dc77cec5c5 parsl.dataflow.memoization INFO: Task 620 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45742020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 620 try 0 launched on executor multi with executor id 1093 parsl.dataflow.dflow INFO: Standard output for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/82d824e6-a68a-4404-a50d-f22cea9ad4c2_calibrate_855136_41.stdout parsl.dataflow.dflow INFO: Standard error for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/82d824e6-a68a-4404-a50d-f22cea9ad4c2_calibrate_855136_41.stderr parsl.dataflow.dflow INFO: Standard output for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/fb0e0002-eb8e-4b0b-b6f9-3314ddc17a01_characterizeImage_855136_41.stdout parsl.dataflow.dflow INFO: Standard error for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/fb0e0002-eb8e-4b0b-b6f9-3314ddc17a01_characterizeImage_855136_41.stderr parsl.dataflow.dflow INFO: Task 601 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e660339cbdae1d2a0881e781075fba01 with result from task 601 parsl.dataflow.dflow DEBUG: Task 601 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 602 has memoization hash c22745e36d5e3028878815d567e53072 parsl.dataflow.memoization INFO: Task 602 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45740cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 602 try 0 launched on executor multi with executor id 1094 parsl.dataflow.dflow INFO: Standard output for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/b4e61284-6164-4eb7-a38d-88c5562d4f5e_calibrate_855136_23.stdout parsl.dataflow.dflow INFO: Standard error for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/b4e61284-6164-4eb7-a38d-88c5562d4f5e_calibrate_855136_23.stderr parsl.dataflow.dflow INFO: Standard output for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/97404417-791e-4ba5-83a1-8532ef9938dd_characterizeImage_855136_23.stdout parsl.dataflow.dflow INFO: Standard error for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/97404417-791e-4ba5-83a1-8532ef9938dd_characterizeImage_855136_23.stderr parsl.dataflow.dflow INFO: Task 826 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b6289d5bbc18069b0c9a400278b721e with result from task 826 parsl.dataflow.dflow DEBUG: Task 826 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 827 has memoization hash 7ecb662f8fa49d3d01790c82ebeaef71 parsl.dataflow.memoization INFO: Task 827 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa456645e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 827 try 0 launched on executor multi with executor id 1095 parsl.dataflow.dflow INFO: Standard output for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/a204a165-b972-45cd-94ef-2f96b61460bb_calibrate_855044_61.stdout parsl.dataflow.dflow INFO: Standard error for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/a204a165-b972-45cd-94ef-2f96b61460bb_calibrate_855044_61.stderr parsl.dataflow.dflow INFO: Standard output for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/36048fa2-8461-4d1a-b30b-664b1ea3e852_characterizeImage_855044_61.stdout parsl.dataflow.dflow INFO: Standard error for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/36048fa2-8461-4d1a-b30b-664b1ea3e852_characterizeImage_855044_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:30 proc_lsst.multi:146 [INFO] found job 16912344 in provider checkpoint proc_lsst.multi INFO: found job 16912344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:30 proc_lsst.multi:146 [INFO] found job 16912345 in provider checkpoint proc_lsst.multi INFO: found job 16912345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:31 proc_lsst.multi:146 [INFO] found job 16912346 in provider checkpoint proc_lsst.multi INFO: found job 16912346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:31 proc_lsst.multi:146 [INFO] found job 16912347 in provider checkpoint proc_lsst.multi INFO: found job 16912347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 764 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc3b9f5ceb82c0eca73c8a75c13dec89 with result from task 764 parsl.dataflow.dflow DEBUG: Task 764 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/d89ca6d1-9a94-4e02-8cc9-d0f0d1f314fb_calibrate_855137_18.stdout parsl.dataflow.dflow INFO: Standard error for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/d89ca6d1-9a94-4e02-8cc9-d0f0d1f314fb_calibrate_855137_18.stderr parsl.dataflow.dflow INFO: Task 308 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f73a2e7a86aeb111dca217b2f9390228 with result from task 308 parsl.dataflow.dflow DEBUG: Task 308 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/65b4adf5-c1fe-4bfa-9f03-96ae43cdb0de_calibrate_855042_4.stdout parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/65b4adf5-c1fe-4bfa-9f03-96ae43cdb0de_calibrate_855042_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:31 proc_lsst.multi:146 [INFO] found job 16912348 in provider checkpoint proc_lsst.multi INFO: found job 16912348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:31 proc_lsst.multi:146 [INFO] found job 16912349 in provider checkpoint proc_lsst.multi INFO: found job 16912349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:32 proc_lsst.multi:146 [INFO] found job 16912350 in provider checkpoint proc_lsst.multi INFO: found job 16912350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:32 proc_lsst.multi:146 [INFO] found job 16912351 in provider checkpoint proc_lsst.multi INFO: found job 16912351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:32 proc_lsst.multi:146 [INFO] found job 16912352 in provider checkpoint proc_lsst.multi INFO: found job 16912352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 343 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c109d37563f4430fdd65f36c3d3682dc with result from task 343 parsl.dataflow.dflow DEBUG: Task 343 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 344 has memoization hash 83acb9ff9967ff7ccee912aaebf226f5 parsl.dataflow.memoization INFO: Task 344 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa46878900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 344 try 0 launched on executor multi with executor id 1096 parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/a55b5269-0ae2-451a-8459-22392d6d7a57_calibrate_855136_59.stdout parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/a55b5269-0ae2-451a-8459-22392d6d7a57_calibrate_855136_59.stderr parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/8aa456b0-a8df-4abc-8aa4-84d0c699ca93_characterizeImage_855136_59.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/8aa456b0-a8df-4abc-8aa4-84d0c699ca93_characterizeImage_855136_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:32 proc_lsst.multi:146 [INFO] found job 16912353 in provider checkpoint proc_lsst.multi INFO: found job 16912353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:32 proc_lsst.multi:146 [INFO] found job 16912354 in provider checkpoint proc_lsst.multi INFO: found job 16912354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:33 proc_lsst.multi:146 [INFO] found job 16912355 in provider checkpoint proc_lsst.multi INFO: found job 16912355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:33 proc_lsst.multi:146 [INFO] found job 16912356 in provider checkpoint proc_lsst.multi INFO: found job 16912356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:33 proc_lsst.multi:146 [INFO] found job 16912359 in provider checkpoint proc_lsst.multi INFO: found job 16912359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:33 proc_lsst.multi:146 [INFO] found job 16912360 in provider checkpoint proc_lsst.multi INFO: found job 16912360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:33 proc_lsst.multi:146 [INFO] found job 16912361 in provider checkpoint proc_lsst.multi INFO: found job 16912361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:34 proc_lsst.multi:146 [INFO] found job 16912362 in provider checkpoint proc_lsst.multi INFO: found job 16912362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:34 proc_lsst.multi:146 [INFO] found job 16912363 in provider checkpoint proc_lsst.multi INFO: found job 16912363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:34 proc_lsst.multi:146 [INFO] found job 16912364 in provider checkpoint proc_lsst.multi INFO: found job 16912364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:34 proc_lsst.multi:146 [INFO] found job 16912365 in provider checkpoint proc_lsst.multi INFO: found job 16912365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:35 proc_lsst.multi:146 [INFO] found job 16912366 in provider checkpoint proc_lsst.multi INFO: found job 16912366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:35 proc_lsst.multi:146 [INFO] found job 16912367 in provider checkpoint proc_lsst.multi INFO: found job 16912367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:35 proc_lsst.multi:146 [INFO] found job 16912368 in provider checkpoint proc_lsst.multi INFO: found job 16912368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:35 proc_lsst.multi:146 [INFO] found job 16912369 in provider checkpoint proc_lsst.multi INFO: found job 16912369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 524 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3b4369f3e85bfa2217eb07e2bba4464 with result from task 524 parsl.dataflow.dflow DEBUG: Task 524 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/7b970726-4aa7-4347-865d-960b4d7a43f1_calibrate_855137_39.stdout parsl.dataflow.dflow INFO: Standard error for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/7b970726-4aa7-4347-865d-960b4d7a43f1_calibrate_855137_39.stderr parsl.dataflow.dflow INFO: Task 640 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a47e49bf392be9628d57e27b602310a with result from task 640 parsl.dataflow.dflow DEBUG: Task 640 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 641 has memoization hash 1a2a4e3af722eb8d2c5ec206579caa96 parsl.dataflow.memoization INFO: Task 641 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 641 try 0 launched on executor multi with executor id 1097 parsl.dataflow.dflow INFO: Standard output for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/f40068e7-3f03-490d-ac99-5c5dd9747d28_calibrate_855138_47.stdout parsl.dataflow.dflow INFO: Standard error for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/f40068e7-3f03-490d-ac99-5c5dd9747d28_calibrate_855138_47.stderr parsl.dataflow.dflow INFO: Standard output for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/fa3f3328-e876-474f-9636-78c578e73243_characterizeImage_855138_47.stdout parsl.dataflow.dflow INFO: Standard error for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/fa3f3328-e876-474f-9636-78c578e73243_characterizeImage_855138_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:35 proc_lsst.multi:146 [INFO] found job 16912370 in provider checkpoint proc_lsst.multi INFO: found job 16912370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 728 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3ee76155564cc3e803d4889fad9f1b6 with result from task 728 parsl.dataflow.dflow DEBUG: Task 728 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/54696afa-5183-4166-9019-dc0856a68b5b_calibrate_855042_41.stdout parsl.dataflow.dflow INFO: Standard error for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/54696afa-5183-4166-9019-dc0856a68b5b_calibrate_855042_41.stderr parsl.dataflow.dflow INFO: Task 1010 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f787a6dbf0872c1b5c6262ea9a372207 with result from task 1010 parsl.dataflow.dflow DEBUG: Task 1010 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/0b9289e1-11ef-498d-bf73-144de8c1c8a5_calibrate_855043_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/0b9289e1-11ef-498d-bf73-144de8c1c8a5_calibrate_855043_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:36 proc_lsst.multi:146 [INFO] found job 16912371 in provider checkpoint proc_lsst.multi INFO: found job 16912371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 668 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9185b08a02566eca5ff96cb520b54e57 with result from task 668 parsl.dataflow.dflow DEBUG: Task 668 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/e14b885e-59a4-4e1d-bf95-b0a929761b58_calibrate_855043_18.stdout parsl.dataflow.dflow INFO: Standard error for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/e14b885e-59a4-4e1d-bf95-b0a929761b58_calibrate_855043_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:36 proc_lsst.multi:146 [INFO] found job 16912372 in provider checkpoint proc_lsst.multi INFO: found job 16912372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:36 proc_lsst.multi:146 [INFO] found job 16912373 in provider checkpoint proc_lsst.multi INFO: found job 16912373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 467 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd57a732f88240f62a3a7d66f6cfc8d7 with result from task 467 parsl.dataflow.dflow DEBUG: Task 467 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/4b4b0e2c-cf0c-484f-b029-9370c91e8647_calibrate_855042_30.stdout parsl.dataflow.dflow INFO: Standard error for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/4b4b0e2c-cf0c-484f-b029-9370c91e8647_calibrate_855042_30.stderr parsl.dataflow.dflow INFO: Task 1036 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7263450d2fcdbe2ebcc04e732a2ddb3f with result from task 1036 parsl.dataflow.dflow DEBUG: Task 1036 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1037 has memoization hash 6f4ce1a9e15b8bb5bceb90b5af61f3cb parsl.dataflow.memoization INFO: Task 1037 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45541da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1037 try 0 launched on executor multi with executor id 1098 parsl.dataflow.dflow INFO: Standard output for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/c09f396d-aa18-498f-b447-57190e92bbfa_calibrate_855136_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/c09f396d-aa18-498f-b447-57190e92bbfa_calibrate_855136_54.stderr parsl.dataflow.dflow INFO: Standard output for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/0c80d029-2691-415f-aec7-435068ce2978_characterizeImage_855136_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/0c80d029-2691-415f-aec7-435068ce2978_characterizeImage_855136_54.stderr parsl.dataflow.dflow INFO: Task 625 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4ca2012112cba1b678751ee713b9c86 with result from task 625 parsl.dataflow.dflow DEBUG: Task 625 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 626 has memoization hash 54e90d71c25895cbb3695234a6c5d9a0 parsl.dataflow.memoization INFO: Task 626 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45742700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 626 try 0 launched on executor multi with executor id 1099 parsl.dataflow.dflow INFO: Standard output for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/9d59d48b-2c8d-42b0-9de5-13c52a5a5eb4_calibrate_855138_34.stdout parsl.dataflow.dflow INFO: Standard error for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/9d59d48b-2c8d-42b0-9de5-13c52a5a5eb4_calibrate_855138_34.stderr parsl.dataflow.dflow INFO: Standard output for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/2ef4106b-603f-403e-a01d-639ad27e9fcb_characterizeImage_855138_34.stdout parsl.dataflow.dflow INFO: Standard error for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/2ef4106b-603f-403e-a01d-639ad27e9fcb_characterizeImage_855138_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:36 proc_lsst.multi:146 [INFO] found job 16912374 in provider checkpoint proc_lsst.multi INFO: found job 16912374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:37 proc_lsst.multi:146 [INFO] found job 16912375 in provider checkpoint proc_lsst.multi INFO: found job 16912375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 628 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc74d3d87b3979c6ba1e602205246fa9 with result from task 628 parsl.dataflow.dflow DEBUG: Task 628 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 629 has memoization hash ec36fb86ee113ddf5d8a830854794597 parsl.dataflow.memoization INFO: Task 629 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45742340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 629 try 0 launched on executor multi with executor id 1100 parsl.dataflow.dflow INFO: Standard output for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/55b29c2f-d454-4733-85aa-3da49c4a837e_calibrate_855137_60.stdout parsl.dataflow.dflow INFO: Standard error for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/55b29c2f-d454-4733-85aa-3da49c4a837e_calibrate_855137_60.stderr parsl.dataflow.dflow INFO: Standard output for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/48f64f52-20be-4871-a120-c66b8049ad9b_characterizeImage_855137_60.stdout parsl.dataflow.dflow INFO: Standard error for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/48f64f52-20be-4871-a120-c66b8049ad9b_characterizeImage_855137_60.stderr parsl.dataflow.dflow INFO: Task 379 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78710f04d2d840def8efd1c3c68bbd57 with result from task 379 parsl.dataflow.dflow DEBUG: Task 379 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 380 has memoization hash 4c9da48de51eb1b75e05ed5f742c71db parsl.dataflow.memoization INFO: Task 380 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4687ade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 380 try 0 launched on executor multi with executor id 1101 parsl.dataflow.dflow INFO: Standard output for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/06995876-75c7-49c0-8bd2-bc100ff07f74_calibrate_855138_14.stdout parsl.dataflow.dflow INFO: Standard error for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/06995876-75c7-49c0-8bd2-bc100ff07f74_calibrate_855138_14.stderr parsl.dataflow.dflow INFO: Standard output for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/9993f394-d214-4112-993b-ba80d1a62cd1_characterizeImage_855138_14.stdout parsl.dataflow.dflow INFO: Standard error for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/9993f394-d214-4112-993b-ba80d1a62cd1_characterizeImage_855138_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:37 proc_lsst.multi:146 [INFO] found job 16912378 in provider checkpoint proc_lsst.multi INFO: found job 16912378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 1060 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d226509f496aa2e7942e9e62821d8534 with result from task 1060 parsl.dataflow.dflow DEBUG: Task 1060 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1061 has memoization hash 3b4378d8678e7ddcc34e9ca0260f4954 parsl.dataflow.memoization INFO: Task 1061 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45542700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1061 try 0 launched on executor multi with executor id 1102 parsl.dataflow.dflow INFO: Standard output for task 1061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/a258ac6d-4271-4582-a73d-0cba0b7d06bf_calibrate_855137_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/a258ac6d-4271-4582-a73d-0cba0b7d06bf_calibrate_855137_11.stderr parsl.dataflow.dflow INFO: Standard output for task 1060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/142c9c74-aff9-4a43-803d-6c967b1c492d_characterizeImage_855137_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/142c9c74-aff9-4a43-803d-6c967b1c492d_characterizeImage_855137_11.stderr parsl.dataflow.dflow INFO: Task 536 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ef0b7589940c9c848411fc55eb7ec4c with result from task 536 parsl.dataflow.dflow DEBUG: Task 536 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/6103caea-c873-4c47-9ab4-5164de9ec718_calibrate_855042_7.stdout parsl.dataflow.dflow INFO: Standard error for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/6103caea-c873-4c47-9ab4-5164de9ec718_calibrate_855042_7.stderr parsl.dataflow.dflow INFO: Task 461 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 28be2d6b73045006503500af207cc7c9 with result from task 461 parsl.dataflow.dflow DEBUG: Task 461 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/fd781cf1-a7ca-4472-b502-31932bf3d056_calibrate_855042_19.stdout parsl.dataflow.dflow INFO: Standard error for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/fd781cf1-a7ca-4472-b502-31932bf3d056_calibrate_855042_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 105 parsl.dataflow.strategy DEBUG: Executor multi has 105 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 105 parsl.dataflow.strategy DEBUG: Executor multi has 105 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.dflow INFO: Task 515 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8c83da50526ebc0bd1ebad29bb55160 with result from task 515 parsl.dataflow.dflow DEBUG: Task 515 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/6384b77d-ecf1-45c1-8e88-323bbb918e9c_calibrate_855043_41.stdout parsl.dataflow.dflow INFO: Standard error for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/6384b77d-ecf1-45c1-8e88-323bbb918e9c_calibrate_855043_41.stderr parsl.dataflow.dflow INFO: Task 377 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6bbd114296741b676ec6533a2bf426cd with result from task 377 parsl.dataflow.dflow DEBUG: Task 377 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/babbdd25-c78e-491d-be3d-7894f9629f68_calibrate_855042_29.stdout parsl.dataflow.dflow INFO: Standard error for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/babbdd25-c78e-491d-be3d-7894f9629f68_calibrate_855042_29.stderr parsl.dataflow.dflow INFO: Task 551 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0883b878762747c05543d28acdda7e2d with result from task 551 parsl.dataflow.dflow DEBUG: Task 551 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/77694236-45c0-4561-a51c-341c1f724b90_calibrate_855044_62.stdout parsl.dataflow.dflow INFO: Standard error for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/77694236-45c0-4561-a51c-341c1f724b90_calibrate_855044_62.stderr parsl.dataflow.dflow INFO: Task 679 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 754a6fb44dc721493ebbaa80ee52fe05 with result from task 679 parsl.dataflow.dflow DEBUG: Task 679 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 680 has memoization hash ba69ed04e221f8df314ca887f107645d parsl.dataflow.memoization INFO: Task 680 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 680 try 0 launched on executor multi with executor id 1103 parsl.dataflow.dflow INFO: Standard output for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/fecf8f25-30c5-4fab-9a7c-4a2aade3de8e_calibrate_855136_11.stdout parsl.dataflow.dflow INFO: Standard error for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/fecf8f25-30c5-4fab-9a7c-4a2aade3de8e_calibrate_855136_11.stderr parsl.dataflow.dflow INFO: Standard output for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/4c80327b-7030-4b69-975e-7e89ddfe4444_characterizeImage_855136_11.stdout parsl.dataflow.dflow INFO: Standard error for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/4c80327b-7030-4b69-975e-7e89ddfe4444_characterizeImage_855136_11.stderr parsl.dataflow.dflow INFO: Task 901 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 278fa7dd2c94bf81ac8715fffff19ffb with result from task 901 parsl.dataflow.dflow DEBUG: Task 901 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 902 has memoization hash 03449868ba1a80ba93ccd1dab427b483 parsl.dataflow.memoization INFO: Task 902 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45498680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 902 try 0 launched on executor multi with executor id 1104 parsl.dataflow.dflow INFO: Standard output for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/694562e6-af95-4bf8-974c-46488a29b189_calibrate_855138_39.stdout parsl.dataflow.dflow INFO: Standard error for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/694562e6-af95-4bf8-974c-46488a29b189_calibrate_855138_39.stderr parsl.dataflow.dflow INFO: Standard output for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/c64fadd5-09f2-4722-a633-50657fd36f15_characterizeImage_855138_39.stdout parsl.dataflow.dflow INFO: Standard error for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/c64fadd5-09f2-4722-a633-50657fd36f15_characterizeImage_855138_39.stderr parsl.dataflow.dflow INFO: Task 530 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ecefbf10bb97baede711c771626c44b with result from task 530 parsl.dataflow.dflow DEBUG: Task 530 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/2d3dd20f-1c88-4d5f-924e-94cffcb81dd3_calibrate_855044_38.stdout parsl.dataflow.dflow INFO: Standard error for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/2d3dd20f-1c88-4d5f-924e-94cffcb81dd3_calibrate_855044_38.stderr parsl.dataflow.dflow INFO: Task 995 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6456f1cdafdffa46e5277fe928bd7f86 with result from task 995 parsl.dataflow.dflow DEBUG: Task 995 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/039074ff-43f8-4f38-a199-0ed3c4a6719e_calibrate_855136_10.stdout parsl.dataflow.dflow INFO: Standard error for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/039074ff-43f8-4f38-a199-0ed3c4a6719e_calibrate_855136_10.stderr parsl.dataflow.dflow INFO: Task 676 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3647bf93b5bf7217d7b7242b2d471a26 with result from task 676 parsl.dataflow.dflow DEBUG: Task 676 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 677 has memoization hash a22458455a17ca1c4371b875f1af1b74 parsl.dataflow.memoization INFO: Task 677 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 677 try 0 launched on executor multi with executor id 1105 parsl.dataflow.dflow INFO: Standard output for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/7d01ce1d-0888-4c65-8883-a781d2888177_calibrate_855044_43.stdout parsl.dataflow.dflow INFO: Standard error for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/7d01ce1d-0888-4c65-8883-a781d2888177_calibrate_855044_43.stderr parsl.dataflow.dflow INFO: Standard output for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/f4941610-c57a-4966-a9bd-e74d8c74465e_characterizeImage_855044_43.stdout parsl.dataflow.dflow INFO: Standard error for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855044/f4941610-c57a-4966-a9bd-e74d8c74465e_characterizeImage_855044_43.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 100 parsl.dataflow.strategy DEBUG: Executor multi has 100 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.dflow INFO: Task 1109 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2092c6e8379eb6e39f1b3d4c24033ca with result from task 1109 parsl.dataflow.dflow DEBUG: Task 1109 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/d74e48bc-a455-4453-877c-7b8075c74c09_calibrate_855138_15.stdout parsl.dataflow.dflow INFO: Standard error for task 1109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/d74e48bc-a455-4453-877c-7b8075c74c09_calibrate_855138_15.stderr parsl.dataflow.dflow INFO: Task 1115 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82323accc9a3f65e1979679da3356d1b with result from task 1115 parsl.dataflow.dflow DEBUG: Task 1115 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/4b4c4f5e-4550-475f-a4fa-813de3fc489e_calibrate_855137_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/4b4c4f5e-4550-475f-a4fa-813de3fc489e_calibrate_855137_56.stderr parsl.dataflow.dflow INFO: Task 887 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b49bd462b5832d9b3feee00fdf18b21 with result from task 887 parsl.dataflow.dflow DEBUG: Task 887 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/18165b7f-aff8-4257-9ab7-4c8cd15da16e_calibrate_855042_53.stdout parsl.dataflow.dflow INFO: Standard error for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/18165b7f-aff8-4257-9ab7-4c8cd15da16e_calibrate_855042_53.stderr parsl.dataflow.dflow INFO: Task 1103 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a47667438e8e4b8acc857599bda52926 with result from task 1103 parsl.dataflow.dflow DEBUG: Task 1103 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/c1743a72-877a-4b81-a62c-5d9cd144c3ee_calibrate_855136_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/c1743a72-877a-4b81-a62c-5d9cd144c3ee_calibrate_855136_18.stderr parsl.dataflow.dflow INFO: Task 521 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 70f41e38648e407d190a7931f5f74ed6 with result from task 521 parsl.dataflow.dflow DEBUG: Task 521 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/47d8eb36-40b9-40df-bf82-dea76991dfd4_calibrate_855137_41.stdout parsl.dataflow.dflow INFO: Standard error for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/47d8eb36-40b9-40df-bf82-dea76991dfd4_calibrate_855137_41.stderr parsl.dataflow.dflow INFO: Task 1090 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db02db5edd399c080aa57d093f7f93b5 with result from task 1090 parsl.dataflow.dflow DEBUG: Task 1090 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1091 has memoization hash c61957eff3dc6bd89650ac22e720ff46 parsl.dataflow.memoization INFO: Task 1091 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45390ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1091 try 0 launched on executor multi with executor id 1106 parsl.dataflow.dflow INFO: Standard output for task 1091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/c7255a50-ff90-4eb2-9259-ed0b9807e9d8_calibrate_855136_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/c7255a50-ff90-4eb2-9259-ed0b9807e9d8_calibrate_855136_17.stderr parsl.dataflow.dflow INFO: Standard output for task 1090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/9ee49d06-ae20-45e4-8a4e-a348c8876e18_characterizeImage_855136_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/9ee49d06-ae20-45e4-8a4e-a348c8876e18_characterizeImage_855136_17.stderr parsl.dataflow.dflow INFO: Task 554 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd26b0a701d7fe6bc00a542e042a1f40 with result from task 554 parsl.dataflow.dflow DEBUG: Task 554 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/a013e6a5-fee0-4622-8014-eeaf767c88cb_calibrate_855044_14.stdout parsl.dataflow.dflow INFO: Standard error for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/a013e6a5-fee0-4622-8014-eeaf767c88cb_calibrate_855044_14.stderr parsl.dataflow.dflow INFO: Task 301 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6092987c05eedc44bd4c7982de8e7db6 with result from task 301 parsl.dataflow.dflow DEBUG: Task 301 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 302 has memoization hash 7d81787f0137e18066e3568677c39589 parsl.dataflow.memoization INFO: Task 302 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4685dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 302 try 0 launched on executor multi with executor id 1107 parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/46bdbe42-9785-40aa-93d2-731ff8b802d5_calibrate_855138_53.stdout parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/46bdbe42-9785-40aa-93d2-731ff8b802d5_calibrate_855138_53.stderr parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/323a04f9-1e39-432a-88b0-219de3272ee3_characterizeImage_855138_53.stdout parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/323a04f9-1e39-432a-88b0-219de3272ee3_characterizeImage_855138_53.stderr parsl.dataflow.dflow INFO: Task 824 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3790f0a75984edc191f16d47764acdd9 with result from task 824 parsl.dataflow.dflow DEBUG: Task 824 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/76c63b73-6669-4a84-a5e7-b55aabf04b86_calibrate_855137_38.stdout parsl.dataflow.dflow INFO: Standard error for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/76c63b73-6669-4a84-a5e7-b55aabf04b86_calibrate_855137_38.stderr 2024-03-08 07:20:47 proc_lsst.multi:146 [INFO] found job 75775 in provider local proc_lsst.multi INFO: found job 75775 in provider local 2024-03-08 07:20:47 proc_lsst.multi:146 [INFO] found job 16912283 in provider astro proc_lsst.multi INFO: found job 16912283 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:47 proc_lsst.multi:146 [INFO] found job 16912284 in provider astro proc_lsst.multi INFO: found job 16912284 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:48 proc_lsst.multi:146 [INFO] found job 16912285 in provider astro proc_lsst.multi INFO: found job 16912285 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 809 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94bb5e638c9ec99ba060d20f28b6bebb with result from task 809 parsl.dataflow.dflow DEBUG: Task 809 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/e9f6cdba-499b-4e69-8076-ac144e700500_calibrate_855137_58.stdout parsl.dataflow.dflow INFO: Standard error for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/e9f6cdba-499b-4e69-8076-ac144e700500_calibrate_855137_58.stderr parsl.dataflow.dflow INFO: Task 953 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3a599732bdf3c2ade51ecda5c66ca08 with result from task 953 parsl.dataflow.dflow DEBUG: Task 953 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/2aa448c5-a320-42fa-a807-548b1814fcb8_calibrate_855042_11.stdout parsl.dataflow.dflow INFO: Standard error for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/2aa448c5-a320-42fa-a807-548b1814fcb8_calibrate_855042_11.stderr parsl.dataflow.dflow INFO: Task 269 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5027c00cde79f77f334374b79512a80 with result from task 269 parsl.dataflow.dflow DEBUG: Task 269 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/a2fdbdea-4929-4673-aaf1-53fe4d7b424e_calibrate_855136_14.stdout parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/a2fdbdea-4929-4673-aaf1-53fe4d7b424e_calibrate_855136_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:49 proc_lsst.multi:146 [INFO] found job 16912286 in provider astro proc_lsst.multi INFO: found job 16912286 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 605 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71407d2e25d11604c551043314f8ac1a with result from task 605 parsl.dataflow.dflow DEBUG: Task 605 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/61cc534e-765c-40d3-8942-1635c2506c09_calibrate_855137_23.stdout parsl.dataflow.dflow INFO: Standard error for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/61cc534e-765c-40d3-8942-1635c2506c09_calibrate_855137_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:49 proc_lsst.multi:146 [INFO] found job 16912287 in provider astro proc_lsst.multi INFO: found job 16912287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 661 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb9d51823482f02f1d66630f437e1565 with result from task 661 parsl.dataflow.dflow DEBUG: Task 661 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 662 has memoization hash 4b385f33f07fa274fc2b6ab694d76a37 parsl.dataflow.memoization INFO: Task 662 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4558f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 662 try 0 launched on executor multi with executor id 1108 parsl.dataflow.dflow INFO: Standard output for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/3d0a89b7-bd01-4b23-8408-12baec45524f_calibrate_855136_47.stdout parsl.dataflow.dflow INFO: Standard error for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/3d0a89b7-bd01-4b23-8408-12baec45524f_calibrate_855136_47.stderr parsl.dataflow.dflow INFO: Standard output for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/5185327c-59b2-4129-9cd2-58854b99ac1c_characterizeImage_855136_47.stdout parsl.dataflow.dflow INFO: Standard error for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/5185327c-59b2-4129-9cd2-58854b99ac1c_characterizeImage_855136_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:49 proc_lsst.multi:146 [INFO] found job 16912288 in provider astro proc_lsst.multi INFO: found job 16912288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:50 proc_lsst.multi:146 [INFO] found job 16912289 in provider astro proc_lsst.multi INFO: found job 16912289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 862 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f0dca257a56beed8b37078782402eaf8 with result from task 862 parsl.dataflow.dflow DEBUG: Task 862 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 863 has memoization hash 7c0c28d28b0487f6a41a0e7acebcc488 parsl.dataflow.memoization INFO: Task 863 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45498fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 863 try 0 launched on executor multi with executor id 1109 parsl.dataflow.dflow INFO: Standard output for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/7d3ab25a-08a2-4d86-8fa0-270e6fde0598_calibrate_855137_53.stdout parsl.dataflow.dflow INFO: Standard error for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/7d3ab25a-08a2-4d86-8fa0-270e6fde0598_calibrate_855137_53.stderr parsl.dataflow.dflow INFO: Standard output for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/a5a9b673-7328-4b5a-a004-daba4a61d9bd_characterizeImage_855137_53.stdout parsl.dataflow.dflow INFO: Standard error for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/a5a9b673-7328-4b5a-a004-daba4a61d9bd_characterizeImage_855137_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:50 proc_lsst.multi:146 [INFO] found job 16912290 in provider astro proc_lsst.multi INFO: found job 16912290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:50 proc_lsst.multi:146 [INFO] found job 16912291 in provider astro proc_lsst.multi INFO: found job 16912291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 218 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d84a293737d18b4d8d632d2933969b01 with result from task 218 parsl.dataflow.dflow DEBUG: Task 218 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/4a798e90-2b57-48bc-82e3-60f42159d00d_calibrate_855043_17.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/4a798e90-2b57-48bc-82e3-60f42159d00d_calibrate_855043_17.stderr parsl.dataflow.dflow INFO: Task 731 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 112d83c3c0eeca083a17449840837cb2 with result from task 731 parsl.dataflow.dflow DEBUG: Task 731 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/dd9afb08-9554-45ee-87d7-10711e1f4c9e_calibrate_855044_47.stdout parsl.dataflow.dflow INFO: Standard error for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/dd9afb08-9554-45ee-87d7-10711e1f4c9e_calibrate_855044_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:50 proc_lsst.multi:146 [INFO] found job 16912292 in provider astro proc_lsst.multi INFO: found job 16912292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:50 proc_lsst.multi:146 [INFO] found job 16912293 in provider astro proc_lsst.multi INFO: found job 16912293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 1064 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 016a18c3e972dbafa52b077d0f70dd54 with result from task 1064 parsl.dataflow.dflow DEBUG: Task 1064 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/b7d9b68d-497f-4ba6-998e-cfb63289d688_calibrate_855044_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/b7d9b68d-497f-4ba6-998e-cfb63289d688_calibrate_855044_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:51 proc_lsst.multi:146 [INFO] found job 16912294 in provider astro proc_lsst.multi INFO: found job 16912294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 1031 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb0de815c72835b00261693303d57a1e with result from task 1031 parsl.dataflow.dflow DEBUG: Task 1031 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/7d9f11ce-4800-43e3-8a87-5ae971b1b8b5_calibrate_855042_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/7d9f11ce-4800-43e3-8a87-5ae971b1b8b5_calibrate_855042_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:51 proc_lsst.multi:146 [INFO] found job 16912295 in provider astro proc_lsst.multi INFO: found job 16912295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:51 proc_lsst.multi:146 [INFO] found job 16912296 in provider astro proc_lsst.multi INFO: found job 16912296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:51 proc_lsst.multi:146 [INFO] found job 16912297 in provider astro proc_lsst.multi INFO: found job 16912297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 686 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 210e3f43bd6c568e03b26b51151e39f7 with result from task 686 parsl.dataflow.dflow DEBUG: Task 686 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/977704e8-a548-4b92-a606-e81a5d29db4a_calibrate_855044_52.stdout parsl.dataflow.dflow INFO: Standard error for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/977704e8-a548-4b92-a606-e81a5d29db4a_calibrate_855044_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:51 proc_lsst.multi:146 [INFO] found job 16912298 in provider astro proc_lsst.multi INFO: found job 16912298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:52 proc_lsst.multi:146 [INFO] found job 16912299 in provider astro proc_lsst.multi INFO: found job 16912299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 719 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d6fdc18a5a55d019f81da244436fb35b with result from task 719 parsl.dataflow.dflow DEBUG: Task 719 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/4f2a0011-b562-4a4e-b59d-89c419d876d3_calibrate_855042_6.stdout parsl.dataflow.dflow INFO: Standard error for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/4f2a0011-b562-4a4e-b59d-89c419d876d3_calibrate_855042_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:52 proc_lsst.multi:146 [INFO] found job 16912300 in provider astro proc_lsst.multi INFO: found job 16912300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:52 proc_lsst.multi:146 [INFO] found job 16912301 in provider astro proc_lsst.multi INFO: found job 16912301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 683 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 26d3cd7c8c12a3cc3f0aa0ae67fae159 with result from task 683 parsl.dataflow.dflow DEBUG: Task 683 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/87745ffd-de8e-41ce-9ef3-cf7c45053892_calibrate_855043_28.stdout parsl.dataflow.dflow INFO: Standard error for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/87745ffd-de8e-41ce-9ef3-cf7c45053892_calibrate_855043_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:52 proc_lsst.multi:146 [INFO] found job 16912302 in provider astro proc_lsst.multi INFO: found job 16912302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:52 proc_lsst.multi:146 [INFO] found job 16912303 in provider astro proc_lsst.multi INFO: found job 16912303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:52 proc_lsst.multi:146 [INFO] found job 16912304 in provider astro proc_lsst.multi INFO: found job 16912304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:53 proc_lsst.multi:146 [INFO] found job 16912305 in provider astro proc_lsst.multi INFO: found job 16912305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:53 proc_lsst.multi:146 [INFO] found job 16912306 in provider astro proc_lsst.multi INFO: found job 16912306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:53 proc_lsst.multi:146 [INFO] found job 16912307 in provider astro proc_lsst.multi INFO: found job 16912307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:53 proc_lsst.multi:146 [INFO] found job 16912308 in provider astro proc_lsst.multi INFO: found job 16912308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:53 proc_lsst.multi:146 [INFO] found job 16912309 in provider astro proc_lsst.multi INFO: found job 16912309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:54 proc_lsst.multi:146 [INFO] found job 16912310 in provider astro proc_lsst.multi INFO: found job 16912310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:54 proc_lsst.multi:146 [INFO] found job 16912311 in provider astro proc_lsst.multi INFO: found job 16912311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 518 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 11fd04f6d9590a27854dde9d43a57c3b with result from task 518 parsl.dataflow.dflow DEBUG: Task 518 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/f45890c3-8a7b-4f26-89f9-cb3255418d02_calibrate_855137_17.stdout parsl.dataflow.dflow INFO: Standard error for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/f45890c3-8a7b-4f26-89f9-cb3255418d02_calibrate_855137_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:54 proc_lsst.multi:146 [INFO] found job 16912312 in provider astro proc_lsst.multi INFO: found job 16912312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:54 proc_lsst.multi:146 [INFO] found job 16912313 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912313 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.dataflow.dflow INFO: Task 473 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3b60a893efb26018bb5d7e711b42da7 with result from task 473 parsl.dataflow.dflow DEBUG: Task 473 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/35ec5dc2-99d7-4484-b545-da5c7642a799_calibrate_855042_42.stdout parsl.dataflow.dflow INFO: Standard error for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/35ec5dc2-99d7-4484-b545-da5c7642a799_calibrate_855042_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:54 proc_lsst.multi:146 [INFO] found job 16912314 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912314 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:55 proc_lsst.multi:146 [INFO] found job 16912315 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912315 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:55 proc_lsst.multi:146 [INFO] found job 16912316 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912316 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.dataflow.dflow INFO: Task 533 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8aafeadf358ddc433f4522b36d110dc8 with result from task 533 parsl.dataflow.dflow DEBUG: Task 533 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/cff4ad65-c038-4e48-87b7-ad5a26b2780d_calibrate_855042_32.stdout parsl.dataflow.dflow INFO: Standard error for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/cff4ad65-c038-4e48-87b7-ad5a26b2780d_calibrate_855042_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:55 proc_lsst.multi:146 [INFO] found job 16912317 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.dataflow.dflow INFO: Task 1046 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2108c9061f80894aa6d1989b23e2766b with result from task 1046 parsl.dataflow.dflow DEBUG: Task 1046 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/7233f557-5c3a-4bbc-9f1a-3d62e748480d_calibrate_855044_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/7233f557-5c3a-4bbc-9f1a-3d62e748480d_calibrate_855044_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:55 proc_lsst.multi:146 [INFO] found job 16912318 in provider escience_a40 proc_lsst.multi INFO: found job 16912318 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 1100 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91dfa564aab0811ea8cc4599e94146de with result from task 1100 parsl.dataflow.dflow DEBUG: Task 1100 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/8dbfe311-2339-4015-8464-a144f2c8f5f3_calibrate_855042_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/8dbfe311-2339-4015-8464-a144f2c8f5f3_calibrate_855042_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:55 proc_lsst.multi:146 [INFO] found job 16912319 in provider escience_a40 proc_lsst.multi INFO: found job 16912319 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:56 proc_lsst.multi:146 [INFO] found job 16912320 in provider escience_a40 proc_lsst.multi INFO: found job 16912320 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:56 proc_lsst.multi:146 [INFO] found job 16912321 in provider escience_a40 proc_lsst.multi INFO: found job 16912321 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 977 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1480303512b8c4b9e4e7718a349f4f02 with result from task 977 parsl.dataflow.dflow DEBUG: Task 977 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/bab9dc8e-91bc-4987-888a-29113d3c1826_calibrate_855044_59.stdout parsl.dataflow.dflow INFO: Standard error for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/bab9dc8e-91bc-4987-888a-29113d3c1826_calibrate_855044_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:56 proc_lsst.multi:146 [INFO] found job 16912322 in provider escience_a40 proc_lsst.multi INFO: found job 16912322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:56 proc_lsst.multi:146 [INFO] found job 16912323 in provider escience_a40 proc_lsst.multi INFO: found job 16912323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:56 proc_lsst.multi:146 [INFO] found job 16912324 in provider escience_a40 proc_lsst.multi INFO: found job 16912324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:56 proc_lsst.multi:146 [INFO] found job 16912325 in provider escience_a40 proc_lsst.multi INFO: found job 16912325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:57 proc_lsst.multi:146 [INFO] found job 16912326 in provider escience_a40 proc_lsst.multi INFO: found job 16912326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:57 proc_lsst.multi:146 [INFO] found job 16912327 in provider escience_a40 proc_lsst.multi INFO: found job 16912327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:57 proc_lsst.multi:146 [INFO] found job 16912328 in provider escience_a40 proc_lsst.multi INFO: found job 16912328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:57 proc_lsst.multi:146 [INFO] found job 16912329 in provider escience_a40 proc_lsst.multi INFO: found job 16912329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:57 proc_lsst.multi:146 [INFO] found job 16912330 in provider escience_a40 proc_lsst.multi INFO: found job 16912330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:58 proc_lsst.multi:146 [INFO] found job 16912331 in provider escience_a40 proc_lsst.multi INFO: found job 16912331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:58 proc_lsst.multi:146 [INFO] found job 16912332 in provider escience_a40 proc_lsst.multi INFO: found job 16912332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:58 proc_lsst.multi:146 [INFO] found job 16912333 in provider escience_a40 proc_lsst.multi INFO: found job 16912333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:58 proc_lsst.multi:146 [INFO] found job 16912334 in provider escience_a40 proc_lsst.multi INFO: found job 16912334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:58 proc_lsst.multi:146 [INFO] found job 16912335 in provider escience_a40 proc_lsst.multi INFO: found job 16912335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 1082 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 107ad66928009f7aa06f0c2c087a53e7 with result from task 1082 parsl.dataflow.dflow DEBUG: Task 1082 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/9a60981d-36a0-415e-a405-a0693a27b647_calibrate_855043_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/9a60981d-36a0-415e-a405-a0693a27b647_calibrate_855043_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:59 proc_lsst.multi:146 [INFO] found job 16912336 in provider escience_a40 proc_lsst.multi INFO: found job 16912336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:59 proc_lsst.multi:146 [INFO] found job 16912337 in provider escience_a40 proc_lsst.multi INFO: found job 16912337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 488 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 93471f5f5ff8b4b77d444511abbb010e with result from task 488 parsl.dataflow.dflow DEBUG: Task 488 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/e3dc2cd0-6814-4a57-90b0-f0a4b0f8e82d_calibrate_855137_35.stdout parsl.dataflow.dflow INFO: Standard error for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/e3dc2cd0-6814-4a57-90b0-f0a4b0f8e82d_calibrate_855137_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:59 proc_lsst.multi:146 [INFO] found job 16912338 in provider checkpoint proc_lsst.multi INFO: found job 16912338 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:59 proc_lsst.multi:146 [INFO] found job 16912339 in provider checkpoint proc_lsst.multi INFO: found job 16912339 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:20:59 proc_lsst.multi:146 [INFO] found job 16912340 in provider checkpoint proc_lsst.multi INFO: found job 16912340 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:00 proc_lsst.multi:146 [INFO] found job 16912341 in provider checkpoint proc_lsst.multi INFO: found job 16912341 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:00 proc_lsst.multi:146 [INFO] found job 16912342 in provider checkpoint proc_lsst.multi INFO: found job 16912342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 812 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 70a89e5206d73f5f485b33ca6705fa3a with result from task 812 parsl.dataflow.dflow DEBUG: Task 812 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/f19fcdcb-73f7-4e9c-8049-530dda9e1beb_calibrate_855044_36.stdout parsl.dataflow.dflow INFO: Standard error for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/f19fcdcb-73f7-4e9c-8049-530dda9e1beb_calibrate_855044_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:00 proc_lsst.multi:146 [INFO] found job 16912343 in provider checkpoint proc_lsst.multi INFO: found job 16912343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 1004 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03673402609d52cf6f8a28a533a5ac04 with result from task 1004 parsl.dataflow.dflow DEBUG: Task 1004 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/f64addfc-6f3e-4ffe-909f-b6a997265498_calibrate_855136_53.stdout parsl.dataflow.dflow INFO: Standard error for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/f64addfc-6f3e-4ffe-909f-b6a997265498_calibrate_855136_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:00 proc_lsst.multi:146 [INFO] found job 16912344 in provider checkpoint proc_lsst.multi INFO: found job 16912344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:00 proc_lsst.multi:146 [INFO] found job 16912345 in provider checkpoint proc_lsst.multi INFO: found job 16912345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:01 proc_lsst.multi:146 [INFO] found job 16912346 in provider checkpoint proc_lsst.multi INFO: found job 16912346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 251 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0c96ed2eac400951fe54832109dba69 with result from task 251 parsl.dataflow.dflow DEBUG: Task 251 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/65984a32-9531-4f10-ad22-71132728401b_calibrate_855136_36.stdout parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/65984a32-9531-4f10-ad22-71132728401b_calibrate_855136_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:01 proc_lsst.multi:146 [INFO] found job 16912347 in provider checkpoint proc_lsst.multi INFO: found job 16912347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:01 proc_lsst.multi:146 [INFO] found job 16912348 in provider checkpoint proc_lsst.multi INFO: found job 16912348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:01 proc_lsst.multi:146 [INFO] found job 16912349 in provider checkpoint proc_lsst.multi INFO: found job 16912349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:01 proc_lsst.multi:146 [INFO] found job 16912350 in provider checkpoint proc_lsst.multi INFO: found job 16912350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:02 proc_lsst.multi:146 [INFO] found job 16912351 in provider checkpoint proc_lsst.multi INFO: found job 16912351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:02 proc_lsst.multi:146 [INFO] found job 16912352 in provider checkpoint proc_lsst.multi INFO: found job 16912352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 974 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 778dbb2c18dea7a783ab1b44ee16419b with result from task 974 parsl.dataflow.dflow DEBUG: Task 974 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/66e5cea6-1084-43f6-8774-32b68fd5d24d_calibrate_855137_10.stdout parsl.dataflow.dflow INFO: Standard error for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/66e5cea6-1084-43f6-8774-32b68fd5d24d_calibrate_855137_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:02 proc_lsst.multi:146 [INFO] found job 16912353 in provider checkpoint proc_lsst.multi INFO: found job 16912353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:02 proc_lsst.multi:146 [INFO] found job 16912354 in provider checkpoint proc_lsst.multi INFO: found job 16912354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:02 proc_lsst.multi:146 [INFO] found job 16912355 in provider checkpoint proc_lsst.multi INFO: found job 16912355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:03 proc_lsst.multi:146 [INFO] found job 16912356 in provider checkpoint proc_lsst.multi INFO: found job 16912356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 797 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e65751d42d3c4bf03077ffb2d4cb0b2 with result from task 797 parsl.dataflow.dflow DEBUG: Task 797 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/f7a34bf7-38c7-42fc-a00e-c0b47d4818b4_calibrate_855136_39.stdout parsl.dataflow.dflow INFO: Standard error for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/f7a34bf7-38c7-42fc-a00e-c0b47d4818b4_calibrate_855136_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:03 proc_lsst.multi:146 [INFO] found job 16912359 in provider checkpoint proc_lsst.multi INFO: found job 16912359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:03 proc_lsst.multi:146 [INFO] found job 16912360 in provider checkpoint proc_lsst.multi INFO: found job 16912360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:03 proc_lsst.multi:146 [INFO] found job 16912361 in provider checkpoint proc_lsst.multi INFO: found job 16912361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 479 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9c414a40c5ec349a3435badd8180c8f with result from task 479 parsl.dataflow.dflow DEBUG: Task 479 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/fcdc7998-0a4f-4f44-a083-fcd9b47b2296_calibrate_855043_60.stdout parsl.dataflow.dflow INFO: Standard error for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/fcdc7998-0a4f-4f44-a083-fcd9b47b2296_calibrate_855043_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:03 proc_lsst.multi:146 [INFO] found job 16912362 in provider checkpoint proc_lsst.multi INFO: found job 16912362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:04 proc_lsst.multi:146 [INFO] found job 16912363 in provider checkpoint proc_lsst.multi INFO: found job 16912363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:04 proc_lsst.multi:146 [INFO] found job 16912364 in provider checkpoint proc_lsst.multi INFO: found job 16912364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:04 proc_lsst.multi:146 [INFO] found job 16912365 in provider checkpoint proc_lsst.multi INFO: found job 16912365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:04 proc_lsst.multi:146 [INFO] found job 16912366 in provider checkpoint proc_lsst.multi INFO: found job 16912366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:04 proc_lsst.multi:146 [INFO] found job 16912367 in provider checkpoint proc_lsst.multi INFO: found job 16912367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 1028 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2033536d3aae7907687bba4833cbdf7a with result from task 1028 parsl.dataflow.dflow DEBUG: Task 1028 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/face6c96-4541-4960-82d3-60560ad8ade3_calibrate_855042_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/face6c96-4541-4960-82d3-60560ad8ade3_calibrate_855042_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:05 proc_lsst.multi:146 [INFO] found job 16912368 in provider checkpoint proc_lsst.multi INFO: found job 16912368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 722 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b314f6b7ca035edad7859d4827822b98 with result from task 722 parsl.dataflow.dflow DEBUG: Task 722 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/d7df3f39-814a-48a4-b2df-5039671ab585_calibrate_855044_9.stdout parsl.dataflow.dflow INFO: Standard error for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/d7df3f39-814a-48a4-b2df-5039671ab585_calibrate_855044_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:05 proc_lsst.multi:146 [INFO] found job 16912369 in provider checkpoint proc_lsst.multi INFO: found job 16912369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:05 proc_lsst.multi:146 [INFO] found job 16912370 in provider checkpoint proc_lsst.multi INFO: found job 16912370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:05 proc_lsst.multi:146 [INFO] found job 16912371 in provider checkpoint proc_lsst.multi INFO: found job 16912371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:05 proc_lsst.multi:146 [INFO] found job 16912372 in provider checkpoint proc_lsst.multi INFO: found job 16912372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:06 proc_lsst.multi:146 [INFO] found job 16912373 in provider checkpoint proc_lsst.multi INFO: found job 16912373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:06 proc_lsst.multi:146 [INFO] found job 16912374 in provider checkpoint proc_lsst.multi INFO: found job 16912374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:06 proc_lsst.multi:146 [INFO] found job 16912375 in provider checkpoint proc_lsst.multi INFO: found job 16912375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:06 proc_lsst.multi:146 [INFO] found job 16912378 in provider checkpoint proc_lsst.multi INFO: found job 16912378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 32 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ebd16fa14adda3a2f820451ce4a934f8 with result from task 32 parsl.dataflow.dflow DEBUG: Task 32 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/9da78cfc-8d0d-4e07-a406-00f3dc8c69ce_calibrate_855043_61.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/9da78cfc-8d0d-4e07-a406-00f3dc8c69ce_calibrate_855043_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 65 parsl.dataflow.strategy DEBUG: Executor multi has 65 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 65 parsl.dataflow.strategy DEBUG: Executor multi has 65 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.dflow INFO: Task 563 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 325867948d6ce9e523a09dbab0a90262 with result from task 563 parsl.dataflow.dflow DEBUG: Task 563 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/270f9954-01b9-465b-bfbd-3ad437337cf1_calibrate_855138_16.stdout parsl.dataflow.dflow INFO: Standard error for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/270f9954-01b9-465b-bfbd-3ad437337cf1_calibrate_855138_16.stderr parsl.dataflow.dflow INFO: Task 1022 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59922ca946b1ab4f3647d334433b9a2f with result from task 1022 parsl.dataflow.dflow DEBUG: Task 1022 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/1ea97871-09e5-4239-afd8-6e1662a19500_calibrate_855042_60.stdout parsl.dataflow.dflow INFO: Standard error for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/1ea97871-09e5-4239-afd8-6e1662a19500_calibrate_855042_60.stderr parsl.dataflow.dflow INFO: Task 1016 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a03452a6d415da69349b496e1e6cbea3 with result from task 1016 parsl.dataflow.dflow DEBUG: Task 1016 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/58ccb9cf-db71-4799-9d17-0420bc7d247f_calibrate_855138_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/58ccb9cf-db71-4799-9d17-0420bc7d247f_calibrate_855138_52.stderr parsl.dataflow.dflow INFO: Task 548 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71cbeaa47a276a24b194051167c37b89 with result from task 548 parsl.dataflow.dflow DEBUG: Task 548 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/d4aa6e07-75c9-4a3d-9bd2-63268d85c214_calibrate_855138_62.stdout parsl.dataflow.dflow INFO: Standard error for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/d4aa6e07-75c9-4a3d-9bd2-63268d85c214_calibrate_855138_62.stderr parsl.dataflow.dflow INFO: Task 1025 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e808fefd4951e35b44512d64e802d2c5 with result from task 1025 parsl.dataflow.dflow DEBUG: Task 1025 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/92099d65-d254-443e-a8dc-3ab48565e3e0_calibrate_855043_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/92099d65-d254-443e-a8dc-3ab48565e3e0_calibrate_855043_11.stderr parsl.dataflow.dflow INFO: Task 941 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0346757f6621fdbb80629da6464fc0d with result from task 941 parsl.dataflow.dflow DEBUG: Task 941 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/d2940fa8-254e-4bdf-a822-0b03f1cf2cee_calibrate_855138_30.stdout parsl.dataflow.dflow INFO: Standard error for task 941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/d2940fa8-254e-4bdf-a822-0b03f1cf2cee_calibrate_855138_30.stderr parsl.dataflow.dflow INFO: Task 1105 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91498483b312b628ffa8b3da897ae412 with result from task 1105 parsl.dataflow.dflow DEBUG: Task 1105 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1106 has memoization hash f610359973cbd75b91ecfd076b2f7360 parsl.dataflow.memoization INFO: Task 1106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45391bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1106 try 0 launched on executor multi with executor id 1110 parsl.dataflow.dflow INFO: Standard output for task 1106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/e6a8cd52-b722-450d-bc60-b92f27524192_calibrate_855138_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/e6a8cd52-b722-450d-bc60-b92f27524192_calibrate_855138_23.stderr parsl.dataflow.dflow INFO: Standard output for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/2b91ebb6-bb74-4077-8d34-fa114cf95594_characterizeImage_855138_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/2b91ebb6-bb74-4077-8d34-fa114cf95594_characterizeImage_855138_23.stderr parsl.dataflow.dflow INFO: Task 1073 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29665f245ace2d43998ec6c450f7cbf4 with result from task 1073 parsl.dataflow.dflow DEBUG: Task 1073 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/83156d7e-3484-479c-8f74-ac6ded3ccae5_calibrate_855136_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/83156d7e-3484-479c-8f74-ac6ded3ccae5_calibrate_855136_24.stderr parsl.dataflow.dflow INFO: Task 1055 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bbb172b2813927a6a5e26b675c58032c with result from task 1055 parsl.dataflow.dflow DEBUG: Task 1055 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/7f9c6961-37fe-4a82-8095-1a064d978b8a_calibrate_855043_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/7f9c6961-37fe-4a82-8095-1a064d978b8a_calibrate_855043_9.stderr parsl.dataflow.dflow INFO: Task 1085 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 34626c122d30bd3b64a4de59b1cfcfcc with result from task 1085 parsl.dataflow.dflow DEBUG: Task 1085 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/371dca44-9cbe-4f7c-a0c7-67664229bb57_calibrate_855136_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/371dca44-9cbe-4f7c-a0c7-67664229bb57_calibrate_855136_19.stderr parsl.dataflow.dflow INFO: Task 326 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8965267c5f84770f82434360901d319 with result from task 326 parsl.dataflow.dflow DEBUG: Task 326 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/b11fb852-8434-4b46-9693-a241b49adaa3_calibrate_855042_52.stdout parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/b11fb852-8434-4b46-9693-a241b49adaa3_calibrate_855042_52.stderr parsl.dataflow.dflow INFO: Task 401 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed26126cd68206c1bb2c92ea6fca44e1 with result from task 401 parsl.dataflow.dflow DEBUG: Task 401 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/4f1275b0-9906-4350-95cf-d3f2b0dc8faf_calibrate_855137_27.stdout parsl.dataflow.dflow INFO: Standard error for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/4f1275b0-9906-4350-95cf-d3f2b0dc8faf_calibrate_855137_27.stderr parsl.dataflow.dflow INFO: Task 1019 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05f9dc1a4d3a653ca851657afd46d72e with result from task 1019 parsl.dataflow.dflow DEBUG: Task 1019 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/a6b7afb7-9a6d-40fe-82ca-6084bc5aa64d_calibrate_855137_24.stdout parsl.dataflow.dflow INFO: Standard error for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/a6b7afb7-9a6d-40fe-82ca-6084bc5aa64d_calibrate_855137_24.stderr parsl.dataflow.dflow INFO: Task 716 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8fcc5ccf9c31d4dc34a1af73783e38bd with result from task 716 parsl.dataflow.dflow DEBUG: Task 716 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/760598ab-0dd3-49a7-aee6-49f159364ebe_calibrate_855137_12.stdout parsl.dataflow.dflow INFO: Standard error for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/760598ab-0dd3-49a7-aee6-49f159364ebe_calibrate_855137_12.stderr parsl.dataflow.dflow INFO: Task 1094 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 245982a3860944a9c31d20ad28df53e7 with result from task 1094 parsl.dataflow.dflow DEBUG: Task 1094 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/9624e48a-6724-4045-867f-0e20d9956ebd_calibrate_855043_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/9624e48a-6724-4045-867f-0e20d9956ebd_calibrate_855043_23.stderr parsl.dataflow.dflow INFO: Task 539 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7bf459a1ff0061cbbffadc785e8b70b4 with result from task 539 parsl.dataflow.dflow DEBUG: Task 539 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/cfe59608-3d54-46b4-9d04-dd622d46cd15_calibrate_855043_10.stdout parsl.dataflow.dflow INFO: Standard error for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/cfe59608-3d54-46b4-9d04-dd622d46cd15_calibrate_855043_10.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 50 parsl.dataflow.strategy DEBUG: Executor multi has 50 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.dflow INFO: Task 623 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27be416df4ec0d828f39ed583412d9e9 with result from task 623 parsl.dataflow.dflow DEBUG: Task 623 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/31cc046b-9a88-4ba3-93da-e960a5114781_calibrate_855136_12.stdout parsl.dataflow.dflow INFO: Standard error for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/31cc046b-9a88-4ba3-93da-e960a5114781_calibrate_855136_12.stderr parsl.dataflow.dflow INFO: Task 674 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55120e13ffe4f66da32ead03db6ef320 with result from task 674 parsl.dataflow.dflow DEBUG: Task 674 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/025748f1-2ac5-41a5-bc12-8af38ef13695_calibrate_855043_16.stdout parsl.dataflow.dflow INFO: Standard error for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/025748f1-2ac5-41a5-bc12-8af38ef13695_calibrate_855043_16.stderr parsl.dataflow.dflow INFO: Task 398 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4212eb15d5fbba9654c9196e64610f6d with result from task 398 parsl.dataflow.dflow DEBUG: Task 398 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/09dd010a-cc95-40b2-9f42-6f7f4e2643ef_calibrate_855138_48.stdout parsl.dataflow.dflow INFO: Standard error for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/09dd010a-cc95-40b2-9f42-6f7f4e2643ef_calibrate_855138_48.stderr parsl.dataflow.dflow INFO: Task 938 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c40ca1ff6bd0278846cbf6d737e9a0c with result from task 938 parsl.dataflow.dflow DEBUG: Task 938 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/a72031b4-6530-40ab-af9a-58beb86e9a64_calibrate_855138_17.stdout parsl.dataflow.dflow INFO: Standard error for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/a72031b4-6530-40ab-af9a-58beb86e9a64_calibrate_855138_17.stderr parsl.dataflow.dflow INFO: Task 1006 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9adb080bb264f3242c46a2fe8e58ddc5 with result from task 1006 parsl.dataflow.dflow DEBUG: Task 1006 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1007 has memoization hash 055d4ed3081b8b185d29396268586295 parsl.dataflow.memoization INFO: Task 1007 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa455400e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1007 try 0 launched on executor multi with executor id 1111 parsl.dataflow.dflow INFO: Standard output for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/8abf6c51-f336-410f-9704-bdde9b9279bb_calibrate_855042_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/8abf6c51-f336-410f-9704-bdde9b9279bb_calibrate_855042_23.stderr parsl.dataflow.dflow INFO: Standard output for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/dca720a9-7944-48cd-90ec-5118c3b99bae_characterizeImage_855042_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/dca720a9-7944-48cd-90ec-5118c3b99bae_characterizeImage_855042_23.stderr parsl.dataflow.dflow INFO: Task 545 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ebd574f2d1213096292dc4b33d573e65 with result from task 545 parsl.dataflow.dflow DEBUG: Task 545 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/ec6dbe93-0eb2-49d0-9ac2-dee432d8571f_calibrate_855138_18.stdout parsl.dataflow.dflow INFO: Standard error for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/ec6dbe93-0eb2-49d0-9ac2-dee432d8571f_calibrate_855138_18.stderr parsl.dataflow.dflow INFO: Task 671 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b3cafef8eb3e375d3bf9a023ea75c56e with result from task 671 parsl.dataflow.dflow DEBUG: Task 671 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/1fe8ddc2-d9e5-4ba4-b064-8bcd3a55c1dd_calibrate_855137_55.stdout parsl.dataflow.dflow INFO: Standard error for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/1fe8ddc2-d9e5-4ba4-b064-8bcd3a55c1dd_calibrate_855137_55.stderr parsl.dataflow.dflow INFO: Task 934 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c32c43637ca0b2584f010862d16b525f with result from task 934 parsl.dataflow.dflow DEBUG: Task 934 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 935 has memoization hash 3de2c8080f072c31b8c7964134809bd0 parsl.dataflow.memoization INFO: Task 935 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa4549b4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 935 try 0 launched on executor multi with executor id 1112 parsl.dataflow.dflow INFO: Standard output for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/f902ce90-d8a0-441a-9486-2322e34a972d_calibrate_855137_54.stdout parsl.dataflow.dflow INFO: Standard error for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/f902ce90-d8a0-441a-9486-2322e34a972d_calibrate_855137_54.stderr parsl.dataflow.dflow INFO: Standard output for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/dcd9040f-7e9a-4980-ad67-e65ecbd76a42_characterizeImage_855137_54.stdout parsl.dataflow.dflow INFO: Standard error for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/dcd9040f-7e9a-4980-ad67-e65ecbd76a42_characterizeImage_855137_54.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 44 parsl.dataflow.strategy DEBUG: Executor multi has 44 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.dflow INFO: Task 986 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3fc07bdc851fd8472ee2779a727380fc with result from task 986 parsl.dataflow.dflow DEBUG: Task 986 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/b7329003-0c45-40c6-a1a6-5aed84f3269d_calibrate_855138_20.stdout parsl.dataflow.dflow INFO: Standard error for task 986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/b7329003-0c45-40c6-a1a6-5aed84f3269d_calibrate_855138_20.stderr parsl.dataflow.dflow INFO: Task 317 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6746aa9e3841bda4f67466702f2b3438 with result from task 317 parsl.dataflow.dflow DEBUG: Task 317 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/c675cc9a-7dd0-4660-b640-18b3fe7b9801_calibrate_855043_26.stdout parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/c675cc9a-7dd0-4660-b640-18b3fe7b9801_calibrate_855043_26.stderr parsl.dataflow.dflow INFO: Task 497 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b43c89683ebf8a25ac41787aa33c56c with result from task 497 parsl.dataflow.dflow DEBUG: Task 497 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/9ba36f31-072d-44c5-ab51-27cf2ba38957_calibrate_855138_51.stdout parsl.dataflow.dflow INFO: Standard error for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/9ba36f31-072d-44c5-ab51-27cf2ba38957_calibrate_855138_51.stderr parsl.dataflow.dflow INFO: Task 176 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4dcb719d08ee950eec049d6677d4c060 with result from task 176 parsl.dataflow.dflow DEBUG: Task 176 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/4c23a0e8-0b53-47ea-b339-66271b6da505_calibrate_855043_59.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855043/4c23a0e8-0b53-47ea-b339-66271b6da505_calibrate_855043_59.stderr parsl.dataflow.dflow INFO: Task 1097 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6423e5287943954fbe03549310a439b9 with result from task 1097 parsl.dataflow.dflow DEBUG: Task 1097 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/a21e61c4-ebac-4c0a-8e59-765180522dd7_calibrate_855137_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/a21e61c4-ebac-4c0a-8e59-765180522dd7_calibrate_855137_37.stderr parsl.dataflow.dflow INFO: Task 1043 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f1ba72f28515fad94cf11cb1d7d248f with result from task 1043 parsl.dataflow.dflow DEBUG: Task 1043 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/4434f278-e59d-4ce6-bdd7-12ca4d2c8aa2_calibrate_855136_16.stdout parsl.dataflow.dflow INFO: Standard error for task 1043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/4434f278-e59d-4ce6-bdd7-12ca4d2c8aa2_calibrate_855136_16.stderr 2024-03-08 07:21:21 proc_lsst.multi:146 [INFO] found job 75775 in provider local proc_lsst.multi INFO: found job 75775 in provider local 2024-03-08 07:21:21 proc_lsst.multi:146 [INFO] found job 16912283 in provider astro proc_lsst.multi INFO: found job 16912283 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 296 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f37f92dddd66500623e8a1f5b1fedbc with result from task 296 parsl.dataflow.dflow DEBUG: Task 296 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/92154ecf-e27c-412f-b4e7-90d757ad1900_calibrate_855138_45.stdout parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/92154ecf-e27c-412f-b4e7-90d757ad1900_calibrate_855138_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:22 proc_lsst.multi:146 [INFO] found job 16912284 in provider astro proc_lsst.multi INFO: found job 16912284 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 632 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dee5d221b908d65e4af8765a4ffc0682 with result from task 632 parsl.dataflow.dflow DEBUG: Task 632 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/fb77bb20-b71d-4460-aa25-ae8900e1b211_calibrate_855044_30.stdout parsl.dataflow.dflow INFO: Standard error for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/fb77bb20-b71d-4460-aa25-ae8900e1b211_calibrate_855044_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:24 proc_lsst.multi:146 [INFO] found job 16912285 in provider astro proc_lsst.multi INFO: found job 16912285 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:24 proc_lsst.multi:146 [INFO] found job 16912286 in provider astro proc_lsst.multi INFO: found job 16912286 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:24 proc_lsst.multi:146 [INFO] found job 16912287 in provider astro proc_lsst.multi INFO: found job 16912287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:24 proc_lsst.multi:146 [INFO] found job 16912288 in provider astro proc_lsst.multi INFO: found job 16912288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 350 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f54ede638daba2636bb73c80bd523bcf with result from task 350 parsl.dataflow.dflow DEBUG: Task 350 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/ed68ce22-c954-4f9b-9ef0-2cd474c58f49_calibrate_855136_30.stdout parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/ed68ce22-c954-4f9b-9ef0-2cd474c58f49_calibrate_855136_30.stderr parsl.dataflow.dflow INFO: Task 1079 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c8fc0b31750f4a6a6ce387e843d9b4c with result from task 1079 parsl.dataflow.dflow DEBUG: Task 1079 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/24eda5b8-551c-41fe-8df2-adc2f078a02d_calibrate_855138_28.stdout parsl.dataflow.dflow INFO: Standard error for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/24eda5b8-551c-41fe-8df2-adc2f078a02d_calibrate_855138_28.stderr parsl.dataflow.dflow INFO: Task 1112 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 863bec0fb881a5cb38460b4b55594ade with result from task 1112 parsl.dataflow.dflow DEBUG: Task 1112 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/52dca954-843c-4429-840b-07f639abf291_calibrate_855138_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/52dca954-843c-4429-840b-07f639abf291_calibrate_855138_54.stderr parsl.dataflow.dflow INFO: Task 356 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 700a9a8efe18706da0f71447cb9854aa with result from task 356 parsl.dataflow.dflow DEBUG: Task 356 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/7caf21bb-73b1-450b-abec-1a3e56063de4_calibrate_855138_10.stdout parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/7caf21bb-73b1-450b-abec-1a3e56063de4_calibrate_855138_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:26 proc_lsst.multi:146 [INFO] found job 16912289 in provider astro proc_lsst.multi INFO: found job 16912289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:26 proc_lsst.multi:146 [INFO] found job 16912290 in provider astro proc_lsst.multi INFO: found job 16912290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 998 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 828ba53f30de359cc4d989fd9be10e2b with result from task 998 parsl.dataflow.dflow DEBUG: Task 998 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/63ab6e21-f386-421f-af25-68be56d5945e_calibrate_855138_26.stdout parsl.dataflow.dflow INFO: Standard error for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/63ab6e21-f386-421f-af25-68be56d5945e_calibrate_855138_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:26 proc_lsst.multi:146 [INFO] found job 16912291 in provider astro proc_lsst.multi INFO: found job 16912291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:27 proc_lsst.multi:146 [INFO] found job 16912292 in provider astro proc_lsst.multi INFO: found job 16912292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:27 proc_lsst.multi:146 [INFO] found job 16912293 in provider astro proc_lsst.multi INFO: found job 16912293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:27 proc_lsst.multi:146 [INFO] found job 16912294 in provider astro proc_lsst.multi INFO: found job 16912294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:27 proc_lsst.multi:146 [INFO] found job 16912295 in provider astro proc_lsst.multi INFO: found job 16912295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:27 proc_lsst.multi:146 [INFO] found job 16912296 in provider astro proc_lsst.multi INFO: found job 16912296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:28 proc_lsst.multi:146 [INFO] found job 16912297 in provider astro proc_lsst.multi INFO: found job 16912297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:28 proc_lsst.multi:146 [INFO] found job 16912298 in provider astro proc_lsst.multi INFO: found job 16912298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:28 proc_lsst.multi:146 [INFO] found job 16912299 in provider astro proc_lsst.multi INFO: found job 16912299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:28 proc_lsst.multi:146 [INFO] found job 16912300 in provider astro proc_lsst.multi INFO: found job 16912300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 1013 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 532e86bcdda6ad9e311ae72d15b17e72 with result from task 1013 parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow DEBUG: Task 1013 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard output for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/7f24bed8-27c7-4c1d-9581-f4927ebb7cc8_calibrate_855138_59.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard error for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/7f24bed8-27c7-4c1d-9581-f4927ebb7cc8_calibrate_855138_59.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:28 proc_lsst.multi:146 [INFO] found job 16912301 in provider astro proc_lsst.multi INFO: found job 16912301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 626 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 54e90d71c25895cbb3695234a6c5d9a0 with result from task 626 parsl.dataflow.dflow DEBUG: Task 626 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/9d59d48b-2c8d-42b0-9de5-13c52a5a5eb4_calibrate_855138_34.stdout parsl.dataflow.dflow INFO: Standard error for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/9d59d48b-2c8d-42b0-9de5-13c52a5a5eb4_calibrate_855138_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:28 proc_lsst.multi:146 [INFO] found job 16912302 in provider astro proc_lsst.multi INFO: found job 16912302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:28 proc_lsst.multi:146 [INFO] found job 16912303 in provider astro proc_lsst.multi INFO: found job 16912303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 944 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09e3ad22cfaa96e09eec63b7aa0962b6 with result from task 944 parsl.dataflow.dflow DEBUG: Task 944 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/948c6828-1762-4a20-862f-5c972183e1e9_calibrate_855138_22.stdout parsl.dataflow.dflow INFO: Standard error for task 944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/948c6828-1762-4a20-862f-5c972183e1e9_calibrate_855138_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:29 proc_lsst.multi:146 [INFO] found job 16912304 in provider astro proc_lsst.multi INFO: found job 16912304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:29 proc_lsst.multi:146 [INFO] found job 16912305 in provider astro proc_lsst.multi INFO: found job 16912305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:29 proc_lsst.multi:146 [INFO] found job 16912306 in provider astro proc_lsst.multi INFO: found job 16912306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:29 proc_lsst.multi:146 [INFO] found job 16912307 in provider astro proc_lsst.multi INFO: found job 16912307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:29 proc_lsst.multi:146 [INFO] found job 16912308 in provider astro proc_lsst.multi INFO: found job 16912308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:29 proc_lsst.multi:146 [INFO] found job 16912309 in provider astro proc_lsst.multi INFO: found job 16912309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:29 proc_lsst.multi:146 [INFO] found job 16912310 in provider astro proc_lsst.multi INFO: found job 16912310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:30 proc_lsst.multi:146 [INFO] found job 16912311 in provider astro proc_lsst.multi INFO: found job 16912311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:30 proc_lsst.multi:146 [INFO] found job 16912312 in provider astro proc_lsst.multi INFO: found job 16912312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:30 proc_lsst.multi:146 [INFO] found job 16912313 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912313 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:30 proc_lsst.multi:146 [INFO] found job 16912314 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912314 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:30 proc_lsst.multi:146 [INFO] found job 16912315 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912315 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:30 proc_lsst.multi:146 [INFO] found job 16912316 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912316 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:30 proc_lsst.multi:146 [INFO] found job 16912317 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:31 proc_lsst.multi:146 [INFO] found job 16912318 in provider escience_a40 proc_lsst.multi INFO: found job 16912318 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 827 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ecb662f8fa49d3d01790c82ebeaef71 with result from task 827 parsl.dataflow.dflow DEBUG: Task 827 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/a204a165-b972-45cd-94ef-2f96b61460bb_calibrate_855044_61.stdout parsl.dataflow.dflow INFO: Standard error for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/a204a165-b972-45cd-94ef-2f96b61460bb_calibrate_855044_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:31 proc_lsst.multi:146 [INFO] found job 16912319 in provider escience_a40 proc_lsst.multi INFO: found job 16912319 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:31 proc_lsst.multi:146 [INFO] found job 16912320 in provider escience_a40 proc_lsst.multi INFO: found job 16912320 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:31 proc_lsst.multi:146 [INFO] found job 16912321 in provider escience_a40 proc_lsst.multi INFO: found job 16912321 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:31 proc_lsst.multi:146 [INFO] found job 16912322 in provider escience_a40 proc_lsst.multi INFO: found job 16912322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:31 proc_lsst.multi:146 [INFO] found job 16912323 in provider escience_a40 proc_lsst.multi INFO: found job 16912323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:32 proc_lsst.multi:146 [INFO] found job 16912324 in provider escience_a40 proc_lsst.multi INFO: found job 16912324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:32 proc_lsst.multi:146 [INFO] found job 16912325 in provider escience_a40 proc_lsst.multi INFO: found job 16912325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:32 proc_lsst.multi:146 [INFO] found job 16912326 in provider escience_a40 proc_lsst.multi INFO: found job 16912326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 641 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a2a4e3af722eb8d2c5ec206579caa96 with result from task 641 parsl.dataflow.dflow DEBUG: Task 641 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/f40068e7-3f03-490d-ac99-5c5dd9747d28_calibrate_855138_47.stdout parsl.dataflow.dflow INFO: Standard error for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/f40068e7-3f03-490d-ac99-5c5dd9747d28_calibrate_855138_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:32 proc_lsst.multi:146 [INFO] found job 16912327 in provider escience_a40 proc_lsst.multi INFO: found job 16912327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:32 proc_lsst.multi:146 [INFO] found job 16912328 in provider escience_a40 proc_lsst.multi INFO: found job 16912328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:32 proc_lsst.multi:146 [INFO] found job 16912329 in provider escience_a40 proc_lsst.multi INFO: found job 16912329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 527 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1114376004fab8ae0ea9fee56b1c2c66 with result from task 527 parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow DEBUG: Task 527 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard output for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/bf1de49c-1552-4027-a8fd-b497150318b7_calibrate_855042_26.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard error for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/bf1de49c-1552-4027-a8fd-b497150318b7_calibrate_855042_26.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:32 proc_lsst.multi:146 [INFO] found job 16912330 in provider escience_a40 proc_lsst.multi INFO: found job 16912330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:33 proc_lsst.multi:146 [INFO] found job 16912331 in provider escience_a40 proc_lsst.multi INFO: found job 16912331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:33 proc_lsst.multi:146 [INFO] found job 16912332 in provider escience_a40 proc_lsst.multi INFO: found job 16912332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:33 proc_lsst.multi:146 [INFO] found job 16912333 in provider escience_a40 proc_lsst.multi INFO: found job 16912333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 902 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03449868ba1a80ba93ccd1dab427b483 with result from task 902 parsl.dataflow.dflow DEBUG: Task 902 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/694562e6-af95-4bf8-974c-46488a29b189_calibrate_855138_39.stdout parsl.dataflow.dflow INFO: Standard error for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/694562e6-af95-4bf8-974c-46488a29b189_calibrate_855138_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:33 proc_lsst.multi:146 [INFO] found job 16912334 in provider escience_a40 proc_lsst.multi INFO: found job 16912334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:33 proc_lsst.multi:146 [INFO] found job 16912335 in provider escience_a40 proc_lsst.multi INFO: found job 16912335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:33 proc_lsst.multi:146 [INFO] found job 16912336 in provider escience_a40 proc_lsst.multi INFO: found job 16912336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.dataflow.dflow INFO: Task 1037 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6f4ce1a9e15b8bb5bceb90b5af61f3cb with result from task 1037 parsl.dataflow.dflow DEBUG: Task 1037 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/c09f396d-aa18-498f-b447-57190e92bbfa_calibrate_855136_54.stdout parsl.dataflow.dflow INFO: Standard error for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/c09f396d-aa18-498f-b447-57190e92bbfa_calibrate_855136_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:34 proc_lsst.multi:146 [INFO] found job 16912337 in provider escience_a40 proc_lsst.multi INFO: found job 16912337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:34 proc_lsst.multi:146 [INFO] found job 16912338 in provider checkpoint proc_lsst.multi INFO: found job 16912338 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 629 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec36fb86ee113ddf5d8a830854794597 with result from task 629 parsl.dataflow.dflow DEBUG: Task 629 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/55b29c2f-d454-4733-85aa-3da49c4a837e_calibrate_855137_60.stdout parsl.dataflow.dflow INFO: Standard error for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/55b29c2f-d454-4733-85aa-3da49c4a837e_calibrate_855137_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:34 proc_lsst.multi:146 [INFO] found job 16912339 in provider checkpoint proc_lsst.multi INFO: found job 16912339 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:34 proc_lsst.multi:146 [INFO] found job 16912340 in provider checkpoint proc_lsst.multi INFO: found job 16912340 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:34 proc_lsst.multi:146 [INFO] found job 16912341 in provider checkpoint proc_lsst.multi INFO: found job 16912341 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:34 proc_lsst.multi:146 [INFO] found job 16912342 in provider checkpoint proc_lsst.multi INFO: found job 16912342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 677 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a22458455a17ca1c4371b875f1af1b74 with result from task 677 parsl.dataflow.dflow DEBUG: Task 677 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/7d01ce1d-0888-4c65-8883-a781d2888177_calibrate_855044_43.stdout parsl.dataflow.dflow INFO: Standard error for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855044/7d01ce1d-0888-4c65-8883-a781d2888177_calibrate_855044_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:34 proc_lsst.multi:146 [INFO] found job 16912343 in provider checkpoint proc_lsst.multi INFO: found job 16912343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:35 proc_lsst.multi:146 [INFO] found job 16912344 in provider checkpoint proc_lsst.multi INFO: found job 16912344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:35 proc_lsst.multi:146 [INFO] found job 16912345 in provider checkpoint proc_lsst.multi INFO: found job 16912345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:35 proc_lsst.multi:146 [INFO] found job 16912346 in provider checkpoint proc_lsst.multi INFO: found job 16912346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:35 proc_lsst.multi:146 [INFO] found job 16912347 in provider checkpoint proc_lsst.multi INFO: found job 16912347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:35 proc_lsst.multi:146 [INFO] found job 16912348 in provider checkpoint proc_lsst.multi INFO: found job 16912348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:35 proc_lsst.multi:146 [INFO] found job 16912349 in provider checkpoint proc_lsst.multi INFO: found job 16912349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:35 proc_lsst.multi:146 [INFO] found job 16912350 in provider checkpoint proc_lsst.multi INFO: found job 16912350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:36 proc_lsst.multi:146 [INFO] found job 16912351 in provider checkpoint proc_lsst.multi INFO: found job 16912351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:36 proc_lsst.multi:146 [INFO] found job 16912352 in provider checkpoint proc_lsst.multi INFO: found job 16912352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:36 proc_lsst.multi:146 [INFO] found job 16912353 in provider checkpoint proc_lsst.multi INFO: found job 16912353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:36 proc_lsst.multi:146 [INFO] found job 16912354 in provider checkpoint proc_lsst.multi INFO: found job 16912354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:36 proc_lsst.multi:146 [INFO] found job 16912355 in provider checkpoint proc_lsst.multi INFO: found job 16912355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:36 proc_lsst.multi:146 [INFO] found job 16912356 in provider checkpoint proc_lsst.multi INFO: found job 16912356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:37 proc_lsst.multi:146 [INFO] found job 16912359 in provider checkpoint proc_lsst.multi INFO: found job 16912359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:37 proc_lsst.multi:146 [INFO] found job 16912360 in provider checkpoint proc_lsst.multi INFO: found job 16912360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:37 proc_lsst.multi:146 [INFO] found job 16912361 in provider checkpoint proc_lsst.multi INFO: found job 16912361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:37 proc_lsst.multi:146 [INFO] found job 16912362 in provider checkpoint proc_lsst.multi INFO: found job 16912362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:37 proc_lsst.multi:146 [INFO] found job 16912363 in provider checkpoint proc_lsst.multi INFO: found job 16912363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:37 proc_lsst.multi:146 [INFO] found job 16912364 in provider checkpoint proc_lsst.multi INFO: found job 16912364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 1061 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b4378d8678e7ddcc34e9ca0260f4954 with result from task 1061 parsl.dataflow.dflow DEBUG: Task 1061 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/a258ac6d-4271-4582-a73d-0cba0b7d06bf_calibrate_855137_11.stdout parsl.dataflow.dflow INFO: Standard error for task 1061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/a258ac6d-4271-4582-a73d-0cba0b7d06bf_calibrate_855137_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:37 proc_lsst.multi:146 [INFO] found job 16912365 in provider checkpoint proc_lsst.multi INFO: found job 16912365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:38 proc_lsst.multi:146 [INFO] found job 16912366 in provider checkpoint proc_lsst.multi INFO: found job 16912366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:38 proc_lsst.multi:146 [INFO] found job 16912367 in provider checkpoint proc_lsst.multi INFO: found job 16912367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:38 proc_lsst.multi:146 [INFO] found job 16912368 in provider checkpoint proc_lsst.multi INFO: found job 16912368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:38 proc_lsst.multi:146 [INFO] found job 16912369 in provider checkpoint proc_lsst.multi INFO: found job 16912369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:38 proc_lsst.multi:146 [INFO] found job 16912370 in provider checkpoint proc_lsst.multi INFO: found job 16912370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:38 proc_lsst.multi:146 [INFO] found job 16912371 in provider checkpoint proc_lsst.multi INFO: found job 16912371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:38 proc_lsst.multi:146 [INFO] found job 16912372 in provider checkpoint proc_lsst.multi INFO: found job 16912372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:39 proc_lsst.multi:146 [INFO] found job 16912373 in provider checkpoint proc_lsst.multi INFO: found job 16912373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:39 proc_lsst.multi:146 [INFO] found job 16912374 in provider checkpoint proc_lsst.multi INFO: found job 16912374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:39 proc_lsst.multi:146 [INFO] found job 16912375 in provider checkpoint proc_lsst.multi INFO: found job 16912375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:39 proc_lsst.multi:146 [INFO] found job 16912378 in provider checkpoint proc_lsst.multi INFO: found job 16912378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 20 parsl.dataflow.strategy DEBUG: Executor multi has 20 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 20 parsl.dataflow.strategy DEBUG: Executor multi has 20 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.dflow INFO: Task 419 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e516114d3a6e84d4b93825dcfa7a1d9 with result from task 419 parsl.dataflow.dflow DEBUG: Task 419 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/a429146b-499d-4e56-9389-a195c74062f3_calibrate_855137_59.stdout parsl.dataflow.dflow INFO: Standard error for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/a429146b-499d-4e56-9389-a195c74062f3_calibrate_855137_59.stderr parsl.dataflow.dflow INFO: Task 1091 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c61957eff3dc6bd89650ac22e720ff46 with result from task 1091 parsl.dataflow.dflow DEBUG: Task 1091 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/c7255a50-ff90-4eb2-9259-ed0b9807e9d8_calibrate_855136_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/c7255a50-ff90-4eb2-9259-ed0b9807e9d8_calibrate_855136_17.stderr parsl.dataflow.dflow INFO: Task 689 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b262dbaba84d36d96fc192a010dcfdba with result from task 689 parsl.dataflow.dflow DEBUG: Task 689 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/71a29078-3ea8-4acd-b460-363b30550f24_calibrate_855136_15.stdout parsl.dataflow.dflow INFO: Standard error for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/71a29078-3ea8-4acd-b460-363b30550f24_calibrate_855136_15.stderr parsl.dataflow.dflow INFO: Task 380 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c9da48de51eb1b75e05ed5f742c71db with result from task 380 parsl.dataflow.dflow DEBUG: Task 380 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/06995876-75c7-49c0-8bd2-bc100ff07f74_calibrate_855138_14.stdout parsl.dataflow.dflow INFO: Standard error for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/06995876-75c7-49c0-8bd2-bc100ff07f74_calibrate_855138_14.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 16 parsl.dataflow.strategy DEBUG: Executor multi has 16 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.dflow INFO: Task 344 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83acb9ff9967ff7ccee912aaebf226f5 with result from task 344 parsl.dataflow.dflow DEBUG: Task 344 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/a55b5269-0ae2-451a-8459-22392d6d7a57_calibrate_855136_59.stdout parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/a55b5269-0ae2-451a-8459-22392d6d7a57_calibrate_855136_59.stderr parsl.dataflow.dflow INFO: Task 863 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c0c28d28b0487f6a41a0e7acebcc488 with result from task 863 parsl.dataflow.dflow DEBUG: Task 863 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/7d3ab25a-08a2-4d86-8fa0-270e6fde0598_calibrate_855137_53.stdout parsl.dataflow.dflow INFO: Standard error for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/7d3ab25a-08a2-4d86-8fa0-270e6fde0598_calibrate_855137_53.stderr parsl.dataflow.dflow INFO: Task 602 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c22745e36d5e3028878815d567e53072 with result from task 602 parsl.dataflow.dflow DEBUG: Task 602 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/b4e61284-6164-4eb7-a38d-88c5562d4f5e_calibrate_855136_23.stdout parsl.dataflow.dflow INFO: Standard error for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/b4e61284-6164-4eb7-a38d-88c5562d4f5e_calibrate_855136_23.stderr parsl.dataflow.dflow INFO: Task 662 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b385f33f07fa274fc2b6ab694d76a37 with result from task 662 parsl.dataflow.dflow DEBUG: Task 662 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/3d0a89b7-bd01-4b23-8408-12baec45524f_calibrate_855136_47.stdout parsl.dataflow.dflow INFO: Standard error for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/3d0a89b7-bd01-4b23-8408-12baec45524f_calibrate_855136_47.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 12 parsl.dataflow.strategy DEBUG: Executor multi has 12 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.dflow INFO: Task 620 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0593c9e6ac0f131b801ba0dc77cec5c5 with result from task 620 parsl.dataflow.dflow DEBUG: Task 620 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/82d824e6-a68a-4404-a50d-f22cea9ad4c2_calibrate_855136_41.stdout parsl.dataflow.dflow INFO: Standard error for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/82d824e6-a68a-4404-a50d-f22cea9ad4c2_calibrate_855136_41.stderr parsl.dataflow.dflow INFO: Task 713 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3321a09d26f1238bed3bce5f0f8e95f5 with result from task 713 parsl.dataflow.dflow DEBUG: Task 713 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/d40504f6-7dc0-4a56-8ade-788d1b3dc53e_calibrate_855138_27.stdout parsl.dataflow.dflow INFO: Standard error for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/d40504f6-7dc0-4a56-8ade-788d1b3dc53e_calibrate_855138_27.stderr parsl.dataflow.dflow INFO: Task 680 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba69ed04e221f8df314ca887f107645d with result from task 680 parsl.dataflow.dflow DEBUG: Task 680 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/fecf8f25-30c5-4fab-9a7c-4a2aade3de8e_calibrate_855136_11.stdout parsl.dataflow.dflow INFO: Standard error for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/fecf8f25-30c5-4fab-9a7c-4a2aade3de8e_calibrate_855136_11.stderr 2024-03-08 07:21:54 proc_lsst.multi:146 [INFO] found job 75775 in provider local proc_lsst.multi INFO: found job 75775 in provider local 2024-03-08 07:21:54 proc_lsst.multi:146 [INFO] found job 16912283 in provider astro proc_lsst.multi INFO: found job 16912283 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:54 proc_lsst.multi:146 [INFO] found job 16912284 in provider astro proc_lsst.multi INFO: found job 16912284 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:55 proc_lsst.multi:146 [INFO] found job 16912285 in provider astro proc_lsst.multi INFO: found job 16912285 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:55 proc_lsst.multi:146 [INFO] found job 16912286 in provider astro proc_lsst.multi INFO: found job 16912286 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:55 proc_lsst.multi:146 [INFO] found job 16912287 in provider astro proc_lsst.multi INFO: found job 16912287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:55 proc_lsst.multi:146 [INFO] found job 16912288 in provider astro proc_lsst.multi INFO: found job 16912288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:55 proc_lsst.multi:146 [INFO] found job 16912289 in provider astro proc_lsst.multi INFO: found job 16912289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:55 proc_lsst.multi:146 [INFO] found job 16912290 in provider astro proc_lsst.multi INFO: found job 16912290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:55 proc_lsst.multi:146 [INFO] found job 16912291 in provider astro proc_lsst.multi INFO: found job 16912291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:56 proc_lsst.multi:146 [INFO] found job 16912292 in provider astro proc_lsst.multi INFO: found job 16912292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:56 proc_lsst.multi:146 [INFO] found job 16912293 in provider astro proc_lsst.multi INFO: found job 16912293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:56 proc_lsst.multi:146 [INFO] found job 16912294 in provider astro proc_lsst.multi INFO: found job 16912294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:56 proc_lsst.multi:146 [INFO] found job 16912295 in provider astro proc_lsst.multi INFO: found job 16912295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:56 proc_lsst.multi:146 [INFO] found job 16912296 in provider astro proc_lsst.multi INFO: found job 16912296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:56 proc_lsst.multi:146 [INFO] found job 16912297 in provider astro proc_lsst.multi INFO: found job 16912297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:56 proc_lsst.multi:146 [INFO] found job 16912298 in provider astro proc_lsst.multi INFO: found job 16912298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:57 proc_lsst.multi:146 [INFO] found job 16912299 in provider astro proc_lsst.multi INFO: found job 16912299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:57 proc_lsst.multi:146 [INFO] found job 16912300 in provider astro proc_lsst.multi INFO: found job 16912300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:57 proc_lsst.multi:146 [INFO] found job 16912301 in provider astro proc_lsst.multi INFO: found job 16912301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:59 proc_lsst.multi:146 [INFO] found job 16912302 in provider astro proc_lsst.multi INFO: found job 16912302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:59 proc_lsst.multi:146 [INFO] found job 16912303 in provider astro proc_lsst.multi INFO: found job 16912303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:59 proc_lsst.multi:146 [INFO] found job 16912304 in provider astro proc_lsst.multi INFO: found job 16912304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:21:59 proc_lsst.multi:146 [INFO] found job 16912305 in provider astro proc_lsst.multi INFO: found job 16912305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 302 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d81787f0137e18066e3568677c39589 with result from task 302 parsl.dataflow.dflow DEBUG: Task 302 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/46bdbe42-9785-40aa-93d2-731ff8b802d5_calibrate_855138_53.stdout parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/46bdbe42-9785-40aa-93d2-731ff8b802d5_calibrate_855138_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:02 proc_lsst.multi:146 [INFO] found job 16912306 in provider astro proc_lsst.multi INFO: found job 16912306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:02 proc_lsst.multi:146 [INFO] found job 16912307 in provider astro proc_lsst.multi INFO: found job 16912307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 1007 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 055d4ed3081b8b185d29396268586295 with result from task 1007 parsl.dataflow.dflow DEBUG: Task 1007 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/8abf6c51-f336-410f-9704-bdde9b9279bb_calibrate_855042_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/8abf6c51-f336-410f-9704-bdde9b9279bb_calibrate_855042_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:02 proc_lsst.multi:146 [INFO] found job 16912308 in provider astro proc_lsst.multi INFO: found job 16912308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:02 proc_lsst.multi:146 [INFO] found job 16912309 in provider astro proc_lsst.multi INFO: found job 16912309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 1106 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f610359973cbd75b91ecfd076b2f7360 with result from task 1106 parsl.dataflow.dflow DEBUG: Task 1106 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/e6a8cd52-b722-450d-bc60-b92f27524192_calibrate_855138_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/e6a8cd52-b722-450d-bc60-b92f27524192_calibrate_855138_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:03 proc_lsst.multi:146 [INFO] found job 16912310 in provider astro proc_lsst.multi INFO: found job 16912310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:03 proc_lsst.multi:146 [INFO] found job 16912311 in provider astro proc_lsst.multi INFO: found job 16912311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:04 proc_lsst.multi:146 [INFO] found job 16912312 in provider astro proc_lsst.multi INFO: found job 16912312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:04 proc_lsst.multi:146 [INFO] found job 16912313 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912313 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:04 proc_lsst.multi:146 [INFO] found job 16912314 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912314 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:04 proc_lsst.multi:146 [INFO] found job 16912315 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912315 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:04 proc_lsst.multi:146 [INFO] found job 16912316 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912316 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:04 proc_lsst.multi:146 [INFO] found job 16912317 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:04 proc_lsst.multi:146 [INFO] found job 16912318 in provider escience_a40 proc_lsst.multi INFO: found job 16912318 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:05 proc_lsst.multi:146 [INFO] found job 16912319 in provider escience_a40 proc_lsst.multi INFO: found job 16912319 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:05 proc_lsst.multi:146 [INFO] found job 16912320 in provider escience_a40 proc_lsst.multi INFO: found job 16912320 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:05 proc_lsst.multi:146 [INFO] found job 16912321 in provider escience_a40 proc_lsst.multi INFO: found job 16912321 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:05 proc_lsst.multi:146 [INFO] found job 16912322 in provider escience_a40 proc_lsst.multi INFO: found job 16912322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:05 proc_lsst.multi:146 [INFO] found job 16912323 in provider escience_a40 proc_lsst.multi INFO: found job 16912323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:05 proc_lsst.multi:146 [INFO] found job 16912324 in provider escience_a40 proc_lsst.multi INFO: found job 16912324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:05 proc_lsst.multi:146 [INFO] found job 16912325 in provider escience_a40 proc_lsst.multi INFO: found job 16912325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:06 proc_lsst.multi:146 [INFO] found job 16912326 in provider escience_a40 proc_lsst.multi INFO: found job 16912326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:06 proc_lsst.multi:146 [INFO] found job 16912327 in provider escience_a40 proc_lsst.multi INFO: found job 16912327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:06 proc_lsst.multi:146 [INFO] found job 16912328 in provider escience_a40 proc_lsst.multi INFO: found job 16912328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:06 proc_lsst.multi:146 [INFO] found job 16912329 in provider escience_a40 proc_lsst.multi INFO: found job 16912329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:06 proc_lsst.multi:146 [INFO] found job 16912330 in provider escience_a40 proc_lsst.multi INFO: found job 16912330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:06 proc_lsst.multi:146 [INFO] found job 16912331 in provider escience_a40 proc_lsst.multi INFO: found job 16912331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:06 proc_lsst.multi:146 [INFO] found job 16912332 in provider escience_a40 proc_lsst.multi INFO: found job 16912332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:07 proc_lsst.multi:146 [INFO] found job 16912333 in provider escience_a40 proc_lsst.multi INFO: found job 16912333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:07 proc_lsst.multi:146 [INFO] found job 16912334 in provider escience_a40 proc_lsst.multi INFO: found job 16912334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:07 proc_lsst.multi:146 [INFO] found job 16912335 in provider escience_a40 proc_lsst.multi INFO: found job 16912335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:07 proc_lsst.multi:146 [INFO] found job 16912336 in provider escience_a40 proc_lsst.multi INFO: found job 16912336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:07 proc_lsst.multi:146 [INFO] found job 16912337 in provider escience_a40 proc_lsst.multi INFO: found job 16912337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:07 proc_lsst.multi:146 [INFO] found job 16912338 in provider checkpoint proc_lsst.multi INFO: found job 16912338 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:07 proc_lsst.multi:146 [INFO] found job 16912339 in provider checkpoint proc_lsst.multi INFO: found job 16912339 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:08 proc_lsst.multi:146 [INFO] found job 16912340 in provider checkpoint proc_lsst.multi INFO: found job 16912340 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:08 proc_lsst.multi:146 [INFO] found job 16912341 in provider checkpoint proc_lsst.multi INFO: found job 16912341 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:08 proc_lsst.multi:146 [INFO] found job 16912342 in provider checkpoint proc_lsst.multi INFO: found job 16912342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 1076 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 707a8e11c554b042364e2bc74d99deba with result from task 1076 parsl.dataflow.dflow DEBUG: Task 1076 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/3248ee6a-5a81-4c7b-ab13-053e91f22e4e_calibrate_855136_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/3248ee6a-5a81-4c7b-ab13-053e91f22e4e_calibrate_855136_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:08 proc_lsst.multi:146 [INFO] found job 16912343 in provider checkpoint proc_lsst.multi INFO: found job 16912343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:08 proc_lsst.multi:146 [INFO] found job 16912344 in provider checkpoint proc_lsst.multi INFO: found job 16912344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:08 proc_lsst.multi:146 [INFO] found job 16912345 in provider checkpoint proc_lsst.multi INFO: found job 16912345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:08 proc_lsst.multi:146 [INFO] found job 16912346 in provider checkpoint proc_lsst.multi INFO: found job 16912346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:09 proc_lsst.multi:146 [INFO] found job 16912347 in provider checkpoint proc_lsst.multi INFO: found job 16912347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:09 proc_lsst.multi:146 [INFO] found job 16912348 in provider checkpoint proc_lsst.multi INFO: found job 16912348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:09 proc_lsst.multi:146 [INFO] found job 16912349 in provider checkpoint proc_lsst.multi INFO: found job 16912349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:09 proc_lsst.multi:146 [INFO] found job 16912350 in provider checkpoint proc_lsst.multi INFO: found job 16912350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:09 proc_lsst.multi:146 [INFO] found job 16912351 in provider checkpoint proc_lsst.multi INFO: found job 16912351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:10 proc_lsst.multi:146 [INFO] found job 16912352 in provider checkpoint proc_lsst.multi INFO: found job 16912352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:10 proc_lsst.multi:146 [INFO] found job 16912353 in provider checkpoint proc_lsst.multi INFO: found job 16912353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:10 proc_lsst.multi:146 [INFO] found job 16912354 in provider checkpoint proc_lsst.multi INFO: found job 16912354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:10 proc_lsst.multi:146 [INFO] found job 16912355 in provider checkpoint proc_lsst.multi INFO: found job 16912355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:10 proc_lsst.multi:146 [INFO] found job 16912356 in provider checkpoint proc_lsst.multi INFO: found job 16912356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:10 proc_lsst.multi:146 [INFO] found job 16912359 in provider checkpoint proc_lsst.multi INFO: found job 16912359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:11 proc_lsst.multi:146 [INFO] found job 16912360 in provider checkpoint proc_lsst.multi INFO: found job 16912360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:11 proc_lsst.multi:146 [INFO] found job 16912361 in provider checkpoint proc_lsst.multi INFO: found job 16912361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:11 proc_lsst.multi:146 [INFO] found job 16912362 in provider checkpoint proc_lsst.multi INFO: found job 16912362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:11 proc_lsst.multi:146 [INFO] found job 16912363 in provider checkpoint proc_lsst.multi INFO: found job 16912363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:11 proc_lsst.multi:146 [INFO] found job 16912364 in provider checkpoint proc_lsst.multi INFO: found job 16912364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:11 proc_lsst.multi:146 [INFO] found job 16912365 in provider checkpoint proc_lsst.multi INFO: found job 16912365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:11 proc_lsst.multi:146 [INFO] found job 16912366 in provider checkpoint proc_lsst.multi INFO: found job 16912366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:12 proc_lsst.multi:146 [INFO] found job 16912367 in provider checkpoint proc_lsst.multi INFO: found job 16912367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:12 proc_lsst.multi:146 [INFO] found job 16912368 in provider checkpoint proc_lsst.multi INFO: found job 16912368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:12 proc_lsst.multi:146 [INFO] found job 16912369 in provider checkpoint proc_lsst.multi INFO: found job 16912369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:12 proc_lsst.multi:146 [INFO] found job 16912370 in provider checkpoint proc_lsst.multi INFO: found job 16912370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:12 proc_lsst.multi:146 [INFO] found job 16912371 in provider checkpoint proc_lsst.multi INFO: found job 16912371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:12 proc_lsst.multi:146 [INFO] found job 16912372 in provider checkpoint proc_lsst.multi INFO: found job 16912372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:13 proc_lsst.multi:146 [INFO] found job 16912373 in provider checkpoint proc_lsst.multi INFO: found job 16912373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:13 proc_lsst.multi:146 [INFO] found job 16912374 in provider checkpoint proc_lsst.multi INFO: found job 16912374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:13 proc_lsst.multi:146 [INFO] found job 16912375 in provider checkpoint proc_lsst.multi INFO: found job 16912375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:13 proc_lsst.multi:146 [INFO] found job 16912378 in provider checkpoint proc_lsst.multi INFO: found job 16912378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 5 parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 5 parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.dflow INFO: Task 10 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c050320f6182e34ae03dceca4bac4a4e with result from task 10 parsl.dataflow.dflow DEBUG: Task 10 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 11 has memoization hash a4a9f346deae42b5ae12a6166cefb0a6 parsl.dataflow.memoization INFO: Task 11 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa467b2a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 11 try 0 launched on executor multi with executor id 1113 parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/1ad05645-519c-4a37-98da-4389fb5e0457_calibrate_855137_61.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/1ad05645-519c-4a37-98da-4389fb5e0457_calibrate_855137_61.stderr parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/8841f386-834a-4d30-8f9b-386cdf36120b_characterizeImage_855137_61.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855137/8841f386-834a-4d30-8f9b-386cdf36120b_characterizeImage_855137_61.stderr parsl.dataflow.dflow INFO: Task 935 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3de2c8080f072c31b8c7964134809bd0 with result from task 935 parsl.dataflow.dflow DEBUG: Task 935 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/f902ce90-d8a0-441a-9486-2322e34a972d_calibrate_855137_54.stdout parsl.dataflow.dflow INFO: Standard error for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/f902ce90-d8a0-441a-9486-2322e34a972d_calibrate_855137_54.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 4 parsl.dataflow.strategy DEBUG: Executor multi has 4 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 4 parsl.dataflow.strategy DEBUG: Executor multi has 4 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 2024-03-08 07:22:28 proc_lsst.multi:146 [INFO] found job 75775 in provider local proc_lsst.multi INFO: found job 75775 in provider local 2024-03-08 07:22:28 proc_lsst.multi:146 [INFO] found job 16912283 in provider astro proc_lsst.multi INFO: found job 16912283 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:28 proc_lsst.multi:146 [INFO] found job 16912284 in provider astro proc_lsst.multi INFO: found job 16912284 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:28 proc_lsst.multi:146 [INFO] found job 16912285 in provider astro proc_lsst.multi INFO: found job 16912285 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:29 proc_lsst.multi:146 [INFO] found job 16912286 in provider astro proc_lsst.multi INFO: found job 16912286 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:29 proc_lsst.multi:146 [INFO] found job 16912287 in provider astro proc_lsst.multi INFO: found job 16912287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:29 proc_lsst.multi:146 [INFO] found job 16912288 in provider astro proc_lsst.multi INFO: found job 16912288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:29 proc_lsst.multi:146 [INFO] found job 16912289 in provider astro proc_lsst.multi INFO: found job 16912289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:29 proc_lsst.multi:146 [INFO] found job 16912290 in provider astro proc_lsst.multi INFO: found job 16912290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:29 proc_lsst.multi:146 [INFO] found job 16912291 in provider astro proc_lsst.multi INFO: found job 16912291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:30 proc_lsst.multi:146 [INFO] found job 16912292 in provider astro proc_lsst.multi INFO: found job 16912292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:30 proc_lsst.multi:146 [INFO] found job 16912293 in provider astro proc_lsst.multi INFO: found job 16912293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:30 proc_lsst.multi:146 [INFO] found job 16912294 in provider astro proc_lsst.multi INFO: found job 16912294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:30 proc_lsst.multi:146 [INFO] found job 16912295 in provider astro proc_lsst.multi INFO: found job 16912295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:30 proc_lsst.multi:146 [INFO] found job 16912296 in provider astro proc_lsst.multi INFO: found job 16912296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:30 proc_lsst.multi:146 [INFO] found job 16912297 in provider astro proc_lsst.multi INFO: found job 16912297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:30 proc_lsst.multi:146 [INFO] found job 16912298 in provider astro proc_lsst.multi INFO: found job 16912298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:31 proc_lsst.multi:146 [INFO] found job 16912299 in provider astro proc_lsst.multi INFO: found job 16912299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:31 proc_lsst.multi:146 [INFO] found job 16912300 in provider astro proc_lsst.multi INFO: found job 16912300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:31 proc_lsst.multi:146 [INFO] found job 16912301 in provider astro proc_lsst.multi INFO: found job 16912301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:31 proc_lsst.multi:146 [INFO] found job 16912302 in provider astro proc_lsst.multi INFO: found job 16912302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:31 proc_lsst.multi:146 [INFO] found job 16912303 in provider astro proc_lsst.multi INFO: found job 16912303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:31 proc_lsst.multi:146 [INFO] found job 16912304 in provider astro proc_lsst.multi INFO: found job 16912304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:31 proc_lsst.multi:146 [INFO] found job 16912305 in provider astro proc_lsst.multi INFO: found job 16912305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:32 proc_lsst.multi:146 [INFO] found job 16912306 in provider astro proc_lsst.multi INFO: found job 16912306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:32 proc_lsst.multi:146 [INFO] found job 16912307 in provider astro proc_lsst.multi INFO: found job 16912307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:32 proc_lsst.multi:146 [INFO] found job 16912308 in provider astro proc_lsst.multi INFO: found job 16912308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:32 proc_lsst.multi:146 [INFO] found job 16912309 in provider astro proc_lsst.multi INFO: found job 16912309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:32 proc_lsst.multi:146 [INFO] found job 16912310 in provider astro proc_lsst.multi INFO: found job 16912310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:32 proc_lsst.multi:146 [INFO] found job 16912311 in provider astro proc_lsst.multi INFO: found job 16912311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:32 proc_lsst.multi:146 [INFO] found job 16912312 in provider astro proc_lsst.multi INFO: found job 16912312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:33 proc_lsst.multi:146 [INFO] found job 16912313 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912313 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:33 proc_lsst.multi:146 [INFO] found job 16912314 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912314 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:33 proc_lsst.multi:146 [INFO] found job 16912315 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912315 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:33 proc_lsst.multi:146 [INFO] found job 16912316 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912316 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:35 proc_lsst.multi:146 [INFO] found job 16912317 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:36 proc_lsst.multi:146 [INFO] found job 16912318 in provider escience_a40 proc_lsst.multi INFO: found job 16912318 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:36 proc_lsst.multi:146 [INFO] found job 16912319 in provider escience_a40 proc_lsst.multi INFO: found job 16912319 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:36 proc_lsst.multi:146 [INFO] found job 16912320 in provider escience_a40 proc_lsst.multi INFO: found job 16912320 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:38 proc_lsst.multi:146 [INFO] found job 16912321 in provider escience_a40 proc_lsst.multi INFO: found job 16912321 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:38 proc_lsst.multi:146 [INFO] found job 16912322 in provider escience_a40 proc_lsst.multi INFO: found job 16912322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:38 proc_lsst.multi:146 [INFO] found job 16912323 in provider escience_a40 proc_lsst.multi INFO: found job 16912323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:38 proc_lsst.multi:146 [INFO] found job 16912324 in provider escience_a40 proc_lsst.multi INFO: found job 16912324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:40 proc_lsst.multi:146 [INFO] found job 16912325 in provider escience_a40 proc_lsst.multi INFO: found job 16912325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:41 proc_lsst.multi:146 [INFO] found job 16912326 in provider escience_a40 proc_lsst.multi INFO: found job 16912326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:41 proc_lsst.multi:146 [INFO] found job 16912327 in provider escience_a40 proc_lsst.multi INFO: found job 16912327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:41 proc_lsst.multi:146 [INFO] found job 16912328 in provider escience_a40 proc_lsst.multi INFO: found job 16912328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:41 proc_lsst.multi:146 [INFO] found job 16912329 in provider escience_a40 proc_lsst.multi INFO: found job 16912329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:41 proc_lsst.multi:146 [INFO] found job 16912330 in provider escience_a40 proc_lsst.multi INFO: found job 16912330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:41 proc_lsst.multi:146 [INFO] found job 16912331 in provider escience_a40 proc_lsst.multi INFO: found job 16912331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:41 proc_lsst.multi:146 [INFO] found job 16912332 in provider escience_a40 proc_lsst.multi INFO: found job 16912332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:42 proc_lsst.multi:146 [INFO] found job 16912333 in provider escience_a40 proc_lsst.multi INFO: found job 16912333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:42 proc_lsst.multi:146 [INFO] found job 16912334 in provider escience_a40 proc_lsst.multi INFO: found job 16912334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:42 proc_lsst.multi:146 [INFO] found job 16912335 in provider escience_a40 proc_lsst.multi INFO: found job 16912335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:42 proc_lsst.multi:146 [INFO] found job 16912336 in provider escience_a40 proc_lsst.multi INFO: found job 16912336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:42 proc_lsst.multi:146 [INFO] found job 16912337 in provider escience_a40 proc_lsst.multi INFO: found job 16912337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:42 proc_lsst.multi:146 [INFO] found job 16912338 in provider checkpoint proc_lsst.multi INFO: found job 16912338 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:42 proc_lsst.multi:146 [INFO] found job 16912339 in provider checkpoint proc_lsst.multi INFO: found job 16912339 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:43 proc_lsst.multi:146 [INFO] found job 16912340 in provider checkpoint proc_lsst.multi INFO: found job 16912340 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:43 proc_lsst.multi:146 [INFO] found job 16912341 in provider checkpoint proc_lsst.multi INFO: found job 16912341 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:43 proc_lsst.multi:146 [INFO] found job 16912342 in provider checkpoint proc_lsst.multi INFO: found job 16912342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:43 proc_lsst.multi:146 [INFO] found job 16912343 in provider checkpoint proc_lsst.multi INFO: found job 16912343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:43 proc_lsst.multi:146 [INFO] found job 16912344 in provider checkpoint proc_lsst.multi INFO: found job 16912344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:43 proc_lsst.multi:146 [INFO] found job 16912345 in provider checkpoint proc_lsst.multi INFO: found job 16912345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:43 proc_lsst.multi:146 [INFO] found job 16912346 in provider checkpoint proc_lsst.multi INFO: found job 16912346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:44 proc_lsst.multi:146 [INFO] found job 16912347 in provider checkpoint proc_lsst.multi INFO: found job 16912347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:44 proc_lsst.multi:146 [INFO] found job 16912348 in provider checkpoint proc_lsst.multi INFO: found job 16912348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:44 proc_lsst.multi:146 [INFO] found job 16912349 in provider checkpoint proc_lsst.multi INFO: found job 16912349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:44 proc_lsst.multi:146 [INFO] found job 16912350 in provider checkpoint proc_lsst.multi INFO: found job 16912350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:44 proc_lsst.multi:146 [INFO] found job 16912351 in provider checkpoint proc_lsst.multi INFO: found job 16912351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:44 proc_lsst.multi:146 [INFO] found job 16912352 in provider checkpoint proc_lsst.multi INFO: found job 16912352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:44 proc_lsst.multi:146 [INFO] found job 16912353 in provider checkpoint proc_lsst.multi INFO: found job 16912353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:45 proc_lsst.multi:146 [INFO] found job 16912354 in provider checkpoint proc_lsst.multi INFO: found job 16912354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:45 proc_lsst.multi:146 [INFO] found job 16912355 in provider checkpoint proc_lsst.multi INFO: found job 16912355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:45 proc_lsst.multi:146 [INFO] found job 16912356 in provider checkpoint proc_lsst.multi INFO: found job 16912356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:45 proc_lsst.multi:146 [INFO] found job 16912359 in provider checkpoint proc_lsst.multi INFO: found job 16912359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:45 proc_lsst.multi:146 [INFO] found job 16912360 in provider checkpoint proc_lsst.multi INFO: found job 16912360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:45 proc_lsst.multi:146 [INFO] found job 16912361 in provider checkpoint proc_lsst.multi INFO: found job 16912361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:45 proc_lsst.multi:146 [INFO] found job 16912362 in provider checkpoint proc_lsst.multi INFO: found job 16912362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:46 proc_lsst.multi:146 [INFO] found job 16912363 in provider checkpoint proc_lsst.multi INFO: found job 16912363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:46 proc_lsst.multi:146 [INFO] found job 16912364 in provider checkpoint proc_lsst.multi INFO: found job 16912364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:46 proc_lsst.multi:146 [INFO] found job 16912365 in provider checkpoint proc_lsst.multi INFO: found job 16912365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:46 proc_lsst.multi:146 [INFO] found job 16912366 in provider checkpoint proc_lsst.multi INFO: found job 16912366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:46 proc_lsst.multi:146 [INFO] found job 16912367 in provider checkpoint proc_lsst.multi INFO: found job 16912367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:46 proc_lsst.multi:146 [INFO] found job 16912368 in provider checkpoint proc_lsst.multi INFO: found job 16912368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:47 proc_lsst.multi:146 [INFO] found job 16912369 in provider checkpoint proc_lsst.multi INFO: found job 16912369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:47 proc_lsst.multi:146 [INFO] found job 16912370 in provider checkpoint proc_lsst.multi INFO: found job 16912370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:47 proc_lsst.multi:146 [INFO] found job 16912371 in provider checkpoint proc_lsst.multi INFO: found job 16912371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:47 proc_lsst.multi:146 [INFO] found job 16912372 in provider checkpoint proc_lsst.multi INFO: found job 16912372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:47 proc_lsst.multi:146 [INFO] found job 16912373 in provider checkpoint proc_lsst.multi INFO: found job 16912373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:47 proc_lsst.multi:146 [INFO] found job 16912374 in provider checkpoint proc_lsst.multi INFO: found job 16912374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:47 proc_lsst.multi:146 [INFO] found job 16912375 in provider checkpoint proc_lsst.multi INFO: found job 16912375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:22:48 proc_lsst.multi:146 [INFO] found job 16912378 in provider checkpoint proc_lsst.multi INFO: found job 16912378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 4 parsl.dataflow.strategy DEBUG: Executor multi has 4 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 4 parsl.dataflow.strategy DEBUG: Executor multi has 4 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 4 parsl.dataflow.strategy DEBUG: Executor multi has 4 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 4 parsl.dataflow.strategy DEBUG: Executor multi has 4 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 2024-03-08 07:23:03 proc_lsst.multi:146 [INFO] found job 75775 in provider local proc_lsst.multi INFO: found job 75775 in provider local 2024-03-08 07:23:03 proc_lsst.multi:146 [INFO] found job 16912283 in provider astro proc_lsst.multi INFO: found job 16912283 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:03 proc_lsst.multi:146 [INFO] found job 16912284 in provider astro proc_lsst.multi INFO: found job 16912284 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:03 proc_lsst.multi:146 [INFO] found job 16912285 in provider astro proc_lsst.multi INFO: found job 16912285 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:03 proc_lsst.multi:146 [INFO] found job 16912286 in provider astro proc_lsst.multi INFO: found job 16912286 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:03 proc_lsst.multi:146 [INFO] found job 16912287 in provider astro proc_lsst.multi INFO: found job 16912287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:03 proc_lsst.multi:146 [INFO] found job 16912288 in provider astro proc_lsst.multi INFO: found job 16912288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:04 proc_lsst.multi:146 [INFO] found job 16912289 in provider astro proc_lsst.multi INFO: found job 16912289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:04 proc_lsst.multi:146 [INFO] found job 16912290 in provider astro proc_lsst.multi INFO: found job 16912290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:04 proc_lsst.multi:146 [INFO] found job 16912291 in provider astro proc_lsst.multi INFO: found job 16912291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:04 proc_lsst.multi:146 [INFO] found job 16912292 in provider astro proc_lsst.multi INFO: found job 16912292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:04 proc_lsst.multi:146 [INFO] found job 16912293 in provider astro proc_lsst.multi INFO: found job 16912293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:04 proc_lsst.multi:146 [INFO] found job 16912294 in provider astro proc_lsst.multi INFO: found job 16912294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:05 proc_lsst.multi:146 [INFO] found job 16912295 in provider astro proc_lsst.multi INFO: found job 16912295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:05 proc_lsst.multi:146 [INFO] found job 16912296 in provider astro proc_lsst.multi INFO: found job 16912296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:05 proc_lsst.multi:146 [INFO] found job 16912297 in provider astro proc_lsst.multi INFO: found job 16912297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:05 proc_lsst.multi:146 [INFO] found job 16912298 in provider astro proc_lsst.multi INFO: found job 16912298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:05 proc_lsst.multi:146 [INFO] found job 16912299 in provider astro proc_lsst.multi INFO: found job 16912299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:05 proc_lsst.multi:146 [INFO] found job 16912300 in provider astro proc_lsst.multi INFO: found job 16912300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:05 proc_lsst.multi:146 [INFO] found job 16912301 in provider astro proc_lsst.multi INFO: found job 16912301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:06 proc_lsst.multi:146 [INFO] found job 16912302 in provider astro proc_lsst.multi INFO: found job 16912302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:06 proc_lsst.multi:146 [INFO] found job 16912303 in provider astro proc_lsst.multi INFO: found job 16912303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:06 proc_lsst.multi:146 [INFO] found job 16912304 in provider astro proc_lsst.multi INFO: found job 16912304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:06 proc_lsst.multi:146 [INFO] found job 16912305 in provider astro proc_lsst.multi INFO: found job 16912305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:06 proc_lsst.multi:146 [INFO] found job 16912306 in provider astro proc_lsst.multi INFO: found job 16912306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:06 proc_lsst.multi:146 [INFO] found job 16912307 in provider astro proc_lsst.multi INFO: found job 16912307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:06 proc_lsst.multi:146 [INFO] found job 16912308 in provider astro proc_lsst.multi INFO: found job 16912308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:07 proc_lsst.multi:146 [INFO] found job 16912309 in provider astro proc_lsst.multi INFO: found job 16912309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:07 proc_lsst.multi:146 [INFO] found job 16912310 in provider astro proc_lsst.multi INFO: found job 16912310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:07 proc_lsst.multi:146 [INFO] found job 16912311 in provider astro proc_lsst.multi INFO: found job 16912311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:07 proc_lsst.multi:146 [INFO] found job 16912312 in provider astro proc_lsst.multi INFO: found job 16912312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:07 proc_lsst.multi:146 [INFO] found job 16912313 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912313 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:07 proc_lsst.multi:146 [INFO] found job 16912314 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912314 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:07 proc_lsst.multi:146 [INFO] found job 16912315 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912315 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:08 proc_lsst.multi:146 [INFO] found job 16912316 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912316 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:08 proc_lsst.multi:146 [INFO] found job 16912317 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:08 proc_lsst.multi:146 [INFO] found job 16912318 in provider escience_a40 proc_lsst.multi INFO: found job 16912318 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:08 proc_lsst.multi:146 [INFO] found job 16912319 in provider escience_a40 proc_lsst.multi INFO: found job 16912319 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:08 proc_lsst.multi:146 [INFO] found job 16912320 in provider escience_a40 proc_lsst.multi INFO: found job 16912320 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:08 proc_lsst.multi:146 [INFO] found job 16912321 in provider escience_a40 proc_lsst.multi INFO: found job 16912321 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:09 proc_lsst.multi:146 [INFO] found job 16912322 in provider escience_a40 proc_lsst.multi INFO: found job 16912322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:09 proc_lsst.multi:146 [INFO] found job 16912323 in provider escience_a40 proc_lsst.multi INFO: found job 16912323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:09 proc_lsst.multi:146 [INFO] found job 16912324 in provider escience_a40 proc_lsst.multi INFO: found job 16912324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:09 proc_lsst.multi:146 [INFO] found job 16912325 in provider escience_a40 proc_lsst.multi INFO: found job 16912325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:09 proc_lsst.multi:146 [INFO] found job 16912326 in provider escience_a40 proc_lsst.multi INFO: found job 16912326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:09 proc_lsst.multi:146 [INFO] found job 16912327 in provider escience_a40 proc_lsst.multi INFO: found job 16912327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:09 proc_lsst.multi:146 [INFO] found job 16912328 in provider escience_a40 proc_lsst.multi INFO: found job 16912328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:10 proc_lsst.multi:146 [INFO] found job 16912329 in provider escience_a40 proc_lsst.multi INFO: found job 16912329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:10 proc_lsst.multi:146 [INFO] found job 16912330 in provider escience_a40 proc_lsst.multi INFO: found job 16912330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:10 proc_lsst.multi:146 [INFO] found job 16912331 in provider escience_a40 proc_lsst.multi INFO: found job 16912331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:10 proc_lsst.multi:146 [INFO] found job 16912332 in provider escience_a40 proc_lsst.multi INFO: found job 16912332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:10 proc_lsst.multi:146 [INFO] found job 16912333 in provider escience_a40 proc_lsst.multi INFO: found job 16912333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:10 proc_lsst.multi:146 [INFO] found job 16912334 in provider escience_a40 proc_lsst.multi INFO: found job 16912334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:12 proc_lsst.multi:146 [INFO] found job 16912335 in provider escience_a40 proc_lsst.multi INFO: found job 16912335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:13 proc_lsst.multi:146 [INFO] found job 16912336 in provider escience_a40 proc_lsst.multi INFO: found job 16912336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:13 proc_lsst.multi:146 [INFO] found job 16912337 in provider escience_a40 proc_lsst.multi INFO: found job 16912337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:13 proc_lsst.multi:146 [INFO] found job 16912338 in provider checkpoint proc_lsst.multi INFO: found job 16912338 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:15 proc_lsst.multi:146 [INFO] found job 16912339 in provider checkpoint proc_lsst.multi INFO: found job 16912339 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:15 proc_lsst.multi:146 [INFO] found job 16912340 in provider checkpoint proc_lsst.multi INFO: found job 16912340 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:15 proc_lsst.multi:146 [INFO] found job 16912341 in provider checkpoint proc_lsst.multi INFO: found job 16912341 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:15 proc_lsst.multi:146 [INFO] found job 16912342 in provider checkpoint proc_lsst.multi INFO: found job 16912342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:16 proc_lsst.multi:146 [INFO] found job 16912343 in provider checkpoint proc_lsst.multi INFO: found job 16912343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:17 proc_lsst.multi:146 [INFO] found job 16912344 in provider checkpoint proc_lsst.multi INFO: found job 16912344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:17 proc_lsst.multi:146 [INFO] found job 16912345 in provider checkpoint proc_lsst.multi INFO: found job 16912345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:17 proc_lsst.multi:146 [INFO] found job 16912346 in provider checkpoint proc_lsst.multi INFO: found job 16912346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:17 proc_lsst.multi:146 [INFO] found job 16912347 in provider checkpoint proc_lsst.multi INFO: found job 16912347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:17 proc_lsst.multi:146 [INFO] found job 16912348 in provider checkpoint proc_lsst.multi INFO: found job 16912348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:17 proc_lsst.multi:146 [INFO] found job 16912349 in provider checkpoint proc_lsst.multi INFO: found job 16912349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:18 proc_lsst.multi:146 [INFO] found job 16912350 in provider checkpoint proc_lsst.multi INFO: found job 16912350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:18 proc_lsst.multi:146 [INFO] found job 16912351 in provider checkpoint proc_lsst.multi INFO: found job 16912351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:18 proc_lsst.multi:146 [INFO] found job 16912352 in provider checkpoint proc_lsst.multi INFO: found job 16912352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:18 proc_lsst.multi:146 [INFO] found job 16912353 in provider checkpoint proc_lsst.multi INFO: found job 16912353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:18 proc_lsst.multi:146 [INFO] found job 16912354 in provider checkpoint proc_lsst.multi INFO: found job 16912354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:18 proc_lsst.multi:146 [INFO] found job 16912355 in provider checkpoint proc_lsst.multi INFO: found job 16912355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:19 proc_lsst.multi:146 [INFO] found job 16912356 in provider checkpoint proc_lsst.multi INFO: found job 16912356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:19 proc_lsst.multi:146 [INFO] found job 16912359 in provider checkpoint proc_lsst.multi INFO: found job 16912359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:19 proc_lsst.multi:146 [INFO] found job 16912360 in provider checkpoint proc_lsst.multi INFO: found job 16912360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:19 proc_lsst.multi:146 [INFO] found job 16912361 in provider checkpoint proc_lsst.multi INFO: found job 16912361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:19 proc_lsst.multi:146 [INFO] found job 16912362 in provider checkpoint proc_lsst.multi INFO: found job 16912362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:19 proc_lsst.multi:146 [INFO] found job 16912363 in provider checkpoint proc_lsst.multi INFO: found job 16912363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:19 proc_lsst.multi:146 [INFO] found job 16912364 in provider checkpoint proc_lsst.multi INFO: found job 16912364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:20 proc_lsst.multi:146 [INFO] found job 16912365 in provider checkpoint proc_lsst.multi INFO: found job 16912365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:20 proc_lsst.multi:146 [INFO] found job 16912366 in provider checkpoint proc_lsst.multi INFO: found job 16912366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:20 proc_lsst.multi:146 [INFO] found job 16912367 in provider checkpoint proc_lsst.multi INFO: found job 16912367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:20 proc_lsst.multi:146 [INFO] found job 16912368 in provider checkpoint proc_lsst.multi INFO: found job 16912368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:20 proc_lsst.multi:146 [INFO] found job 16912369 in provider checkpoint proc_lsst.multi INFO: found job 16912369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:20 proc_lsst.multi:146 [INFO] found job 16912370 in provider checkpoint proc_lsst.multi INFO: found job 16912370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:21 proc_lsst.multi:146 [INFO] found job 16912371 in provider checkpoint proc_lsst.multi INFO: found job 16912371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:21 proc_lsst.multi:146 [INFO] found job 16912372 in provider checkpoint proc_lsst.multi INFO: found job 16912372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:21 proc_lsst.multi:146 [INFO] found job 16912373 in provider checkpoint proc_lsst.multi INFO: found job 16912373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:21 proc_lsst.multi:146 [INFO] found job 16912374 in provider checkpoint proc_lsst.multi INFO: found job 16912374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:21 proc_lsst.multi:146 [INFO] found job 16912375 in provider checkpoint proc_lsst.multi INFO: found job 16912375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:21 proc_lsst.multi:146 [INFO] found job 16912378 in provider checkpoint proc_lsst.multi INFO: found job 16912378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 4 parsl.dataflow.strategy DEBUG: Executor multi has 4 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 4 parsl.dataflow.strategy DEBUG: Executor multi has 4 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.dflow INFO: Task 11 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4a9f346deae42b5ae12a6166cefb0a6 with result from task 11 parsl.dataflow.dflow DEBUG: Task 11 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/1ad05645-519c-4a37-98da-4389fb5e0457_calibrate_855137_61.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855137/1ad05645-519c-4a37-98da-4389fb5e0457_calibrate_855137_61.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.dflow INFO: Task 1039 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1ca11e370a19e38dbaa3022b316e021 with result from task 1039 parsl.dataflow.dflow DEBUG: Task 1039 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1040 has memoization hash 64945d563079654efc7edf65a5ed36e5 parsl.dataflow.memoization INFO: Task 1040 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45542200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1040 try 0 launched on executor multi with executor id 1114 parsl.dataflow.dflow INFO: Standard output for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/9b04132a-2510-4c21-ba68-f822b265e350_calibrate_855138_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/9b04132a-2510-4c21-ba68-f822b265e350_calibrate_855138_61.stderr parsl.dataflow.dflow INFO: Standard output for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/226247c2-de41-4cd1-8d17-aec7cc613876_characterizeImage_855138_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855138/226247c2-de41-4cd1-8d17-aec7cc613876_characterizeImage_855138_61.stderr 2024-03-08 07:23:36 proc_lsst.multi:146 [INFO] found job 75775 in provider local proc_lsst.multi INFO: found job 75775 in provider local 2024-03-08 07:23:37 proc_lsst.multi:146 [INFO] found job 16912283 in provider astro proc_lsst.multi INFO: found job 16912283 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:37 proc_lsst.multi:146 [INFO] found job 16912284 in provider astro proc_lsst.multi INFO: found job 16912284 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:37 proc_lsst.multi:146 [INFO] found job 16912285 in provider astro proc_lsst.multi INFO: found job 16912285 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:37 proc_lsst.multi:146 [INFO] found job 16912286 in provider astro proc_lsst.multi INFO: found job 16912286 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:37 proc_lsst.multi:146 [INFO] found job 16912287 in provider astro proc_lsst.multi INFO: found job 16912287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:37 proc_lsst.multi:146 [INFO] found job 16912288 in provider astro proc_lsst.multi INFO: found job 16912288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:37 proc_lsst.multi:146 [INFO] found job 16912289 in provider astro proc_lsst.multi INFO: found job 16912289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:38 proc_lsst.multi:146 [INFO] found job 16912290 in provider astro proc_lsst.multi INFO: found job 16912290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:38 proc_lsst.multi:146 [INFO] found job 16912291 in provider astro proc_lsst.multi INFO: found job 16912291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:38 proc_lsst.multi:146 [INFO] found job 16912292 in provider astro proc_lsst.multi INFO: found job 16912292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:38 proc_lsst.multi:146 [INFO] found job 16912293 in provider astro proc_lsst.multi INFO: found job 16912293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:38 proc_lsst.multi:146 [INFO] found job 16912294 in provider astro proc_lsst.multi INFO: found job 16912294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:38 proc_lsst.multi:146 [INFO] found job 16912295 in provider astro proc_lsst.multi INFO: found job 16912295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:38 proc_lsst.multi:146 [INFO] found job 16912296 in provider astro proc_lsst.multi INFO: found job 16912296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:39 proc_lsst.multi:146 [INFO] found job 16912297 in provider astro proc_lsst.multi INFO: found job 16912297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:39 proc_lsst.multi:146 [INFO] found job 16912298 in provider astro proc_lsst.multi INFO: found job 16912298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:39 proc_lsst.multi:146 [INFO] found job 16912299 in provider astro proc_lsst.multi INFO: found job 16912299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:39 proc_lsst.multi:146 [INFO] found job 16912300 in provider astro proc_lsst.multi INFO: found job 16912300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:39 proc_lsst.multi:146 [INFO] found job 16912301 in provider astro proc_lsst.multi INFO: found job 16912301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:39 proc_lsst.multi:146 [INFO] found job 16912302 in provider astro proc_lsst.multi INFO: found job 16912302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:40 proc_lsst.multi:146 [INFO] found job 16912303 in provider astro proc_lsst.multi INFO: found job 16912303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:40 proc_lsst.multi:146 [INFO] found job 16912304 in provider astro proc_lsst.multi INFO: found job 16912304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:40 proc_lsst.multi:146 [INFO] found job 16912305 in provider astro proc_lsst.multi INFO: found job 16912305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:40 proc_lsst.multi:146 [INFO] found job 16912306 in provider astro proc_lsst.multi INFO: found job 16912306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:40 proc_lsst.multi:146 [INFO] found job 16912307 in provider astro proc_lsst.multi INFO: found job 16912307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:40 proc_lsst.multi:146 [INFO] found job 16912308 in provider astro proc_lsst.multi INFO: found job 16912308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:40 proc_lsst.multi:146 [INFO] found job 16912309 in provider astro proc_lsst.multi INFO: found job 16912309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:41 proc_lsst.multi:146 [INFO] found job 16912310 in provider astro proc_lsst.multi INFO: found job 16912310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:41 proc_lsst.multi:146 [INFO] found job 16912311 in provider astro proc_lsst.multi INFO: found job 16912311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:41 proc_lsst.multi:146 [INFO] found job 16912312 in provider astro proc_lsst.multi INFO: found job 16912312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:41 proc_lsst.multi:146 [INFO] found job 16912313 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912313 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:41 proc_lsst.multi:146 [INFO] found job 16912314 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912314 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:41 proc_lsst.multi:146 [INFO] found job 16912315 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912315 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:41 proc_lsst.multi:146 [INFO] found job 16912316 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912316 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:42 proc_lsst.multi:146 [INFO] found job 16912317 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:42 proc_lsst.multi:146 [INFO] found job 16912318 in provider escience_a40 proc_lsst.multi INFO: found job 16912318 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:42 proc_lsst.multi:146 [INFO] found job 16912319 in provider escience_a40 proc_lsst.multi INFO: found job 16912319 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:42 proc_lsst.multi:146 [INFO] found job 16912320 in provider escience_a40 proc_lsst.multi INFO: found job 16912320 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:42 proc_lsst.multi:146 [INFO] found job 16912321 in provider escience_a40 proc_lsst.multi INFO: found job 16912321 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:42 proc_lsst.multi:146 [INFO] found job 16912322 in provider escience_a40 proc_lsst.multi INFO: found job 16912322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:42 proc_lsst.multi:146 [INFO] found job 16912323 in provider escience_a40 proc_lsst.multi INFO: found job 16912323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:43 proc_lsst.multi:146 [INFO] found job 16912324 in provider escience_a40 proc_lsst.multi INFO: found job 16912324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:43 proc_lsst.multi:146 [INFO] found job 16912325 in provider escience_a40 proc_lsst.multi INFO: found job 16912325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:43 proc_lsst.multi:146 [INFO] found job 16912326 in provider escience_a40 proc_lsst.multi INFO: found job 16912326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:43 proc_lsst.multi:146 [INFO] found job 16912327 in provider escience_a40 proc_lsst.multi INFO: found job 16912327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:43 proc_lsst.multi:146 [INFO] found job 16912328 in provider escience_a40 proc_lsst.multi INFO: found job 16912328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:43 proc_lsst.multi:146 [INFO] found job 16912329 in provider escience_a40 proc_lsst.multi INFO: found job 16912329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:43 proc_lsst.multi:146 [INFO] found job 16912330 in provider escience_a40 proc_lsst.multi INFO: found job 16912330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:44 proc_lsst.multi:146 [INFO] found job 16912331 in provider escience_a40 proc_lsst.multi INFO: found job 16912331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:44 proc_lsst.multi:146 [INFO] found job 16912332 in provider escience_a40 proc_lsst.multi INFO: found job 16912332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:44 proc_lsst.multi:146 [INFO] found job 16912333 in provider escience_a40 proc_lsst.multi INFO: found job 16912333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:44 proc_lsst.multi:146 [INFO] found job 16912334 in provider escience_a40 proc_lsst.multi INFO: found job 16912334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:45 proc_lsst.multi:146 [INFO] found job 16912335 in provider escience_a40 proc_lsst.multi INFO: found job 16912335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:45 proc_lsst.multi:146 [INFO] found job 16912336 in provider escience_a40 proc_lsst.multi INFO: found job 16912336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:45 proc_lsst.multi:146 [INFO] found job 16912337 in provider escience_a40 proc_lsst.multi INFO: found job 16912337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:45 proc_lsst.multi:146 [INFO] found job 16912338 in provider checkpoint proc_lsst.multi INFO: found job 16912338 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:45 proc_lsst.multi:146 [INFO] found job 16912339 in provider checkpoint proc_lsst.multi INFO: found job 16912339 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:45 proc_lsst.multi:146 [INFO] found job 16912340 in provider checkpoint proc_lsst.multi INFO: found job 16912340 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:45 proc_lsst.multi:146 [INFO] found job 16912341 in provider checkpoint proc_lsst.multi INFO: found job 16912341 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:46 proc_lsst.multi:146 [INFO] found job 16912342 in provider checkpoint proc_lsst.multi INFO: found job 16912342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:46 proc_lsst.multi:146 [INFO] found job 16912343 in provider checkpoint proc_lsst.multi INFO: found job 16912343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:46 proc_lsst.multi:146 [INFO] found job 16912344 in provider checkpoint proc_lsst.multi INFO: found job 16912344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:46 proc_lsst.multi:146 [INFO] found job 16912345 in provider checkpoint proc_lsst.multi INFO: found job 16912345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:46 proc_lsst.multi:146 [INFO] found job 16912346 in provider checkpoint proc_lsst.multi INFO: found job 16912346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:46 proc_lsst.multi:146 [INFO] found job 16912347 in provider checkpoint proc_lsst.multi INFO: found job 16912347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:49 proc_lsst.multi:146 [INFO] found job 16912348 in provider checkpoint proc_lsst.multi INFO: found job 16912348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:49 proc_lsst.multi:146 [INFO] found job 16912349 in provider checkpoint proc_lsst.multi INFO: found job 16912349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:49 proc_lsst.multi:146 [INFO] found job 16912350 in provider checkpoint proc_lsst.multi INFO: found job 16912350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:49 proc_lsst.multi:146 [INFO] found job 16912351 in provider checkpoint proc_lsst.multi INFO: found job 16912351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:51 proc_lsst.multi:146 [INFO] found job 16912352 in provider checkpoint proc_lsst.multi INFO: found job 16912352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:51 proc_lsst.multi:146 [INFO] found job 16912353 in provider checkpoint proc_lsst.multi INFO: found job 16912353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:51 proc_lsst.multi:146 [INFO] found job 16912354 in provider checkpoint proc_lsst.multi INFO: found job 16912354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:52 proc_lsst.multi:146 [INFO] found job 16912355 in provider checkpoint proc_lsst.multi INFO: found job 16912355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:54 proc_lsst.multi:146 [INFO] found job 16912356 in provider checkpoint proc_lsst.multi INFO: found job 16912356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:54 proc_lsst.multi:146 [INFO] found job 16912359 in provider checkpoint proc_lsst.multi INFO: found job 16912359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:54 proc_lsst.multi:146 [INFO] found job 16912360 in provider checkpoint proc_lsst.multi INFO: found job 16912360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:54 proc_lsst.multi:146 [INFO] found job 16912361 in provider checkpoint proc_lsst.multi INFO: found job 16912361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:54 proc_lsst.multi:146 [INFO] found job 16912362 in provider checkpoint proc_lsst.multi INFO: found job 16912362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:54 proc_lsst.multi:146 [INFO] found job 16912363 in provider checkpoint proc_lsst.multi INFO: found job 16912363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:55 proc_lsst.multi:146 [INFO] found job 16912364 in provider checkpoint proc_lsst.multi INFO: found job 16912364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:55 proc_lsst.multi:146 [INFO] found job 16912365 in provider checkpoint proc_lsst.multi INFO: found job 16912365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:55 proc_lsst.multi:146 [INFO] found job 16912366 in provider checkpoint proc_lsst.multi INFO: found job 16912366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:55 proc_lsst.multi:146 [INFO] found job 16912367 in provider checkpoint proc_lsst.multi INFO: found job 16912367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:55 proc_lsst.multi:146 [INFO] found job 16912368 in provider checkpoint proc_lsst.multi INFO: found job 16912368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:55 proc_lsst.multi:146 [INFO] found job 16912369 in provider checkpoint proc_lsst.multi INFO: found job 16912369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:55 proc_lsst.multi:146 [INFO] found job 16912370 in provider checkpoint proc_lsst.multi INFO: found job 16912370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:56 proc_lsst.multi:146 [INFO] found job 16912371 in provider checkpoint proc_lsst.multi INFO: found job 16912371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:56 proc_lsst.multi:146 [INFO] found job 16912372 in provider checkpoint proc_lsst.multi INFO: found job 16912372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:56 proc_lsst.multi:146 [INFO] found job 16912373 in provider checkpoint proc_lsst.multi INFO: found job 16912373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:56 proc_lsst.multi:146 [INFO] found job 16912374 in provider checkpoint proc_lsst.multi INFO: found job 16912374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:56 proc_lsst.multi:146 [INFO] found job 16912375 in provider checkpoint proc_lsst.multi INFO: found job 16912375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:23:56 proc_lsst.multi:146 [INFO] found job 16912378 in provider checkpoint proc_lsst.multi INFO: found job 16912378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 2024-03-08 07:24:11 proc_lsst.multi:146 [INFO] found job 75775 in provider local proc_lsst.multi INFO: found job 75775 in provider local 2024-03-08 07:24:11 proc_lsst.multi:146 [INFO] found job 16912283 in provider astro proc_lsst.multi INFO: found job 16912283 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:12 proc_lsst.multi:146 [INFO] found job 16912284 in provider astro proc_lsst.multi INFO: found job 16912284 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:12 proc_lsst.multi:146 [INFO] found job 16912285 in provider astro proc_lsst.multi INFO: found job 16912285 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:12 proc_lsst.multi:146 [INFO] found job 16912286 in provider astro proc_lsst.multi INFO: found job 16912286 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:12 proc_lsst.multi:146 [INFO] found job 16912287 in provider astro proc_lsst.multi INFO: found job 16912287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:12 proc_lsst.multi:146 [INFO] found job 16912288 in provider astro proc_lsst.multi INFO: found job 16912288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:12 proc_lsst.multi:146 [INFO] found job 16912289 in provider astro proc_lsst.multi INFO: found job 16912289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:13 proc_lsst.multi:146 [INFO] found job 16912290 in provider astro proc_lsst.multi INFO: found job 16912290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:13 proc_lsst.multi:146 [INFO] found job 16912291 in provider astro proc_lsst.multi INFO: found job 16912291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:13 proc_lsst.multi:146 [INFO] found job 16912292 in provider astro proc_lsst.multi INFO: found job 16912292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:13 proc_lsst.multi:146 [INFO] found job 16912293 in provider astro proc_lsst.multi INFO: found job 16912293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:13 proc_lsst.multi:146 [INFO] found job 16912294 in provider astro proc_lsst.multi INFO: found job 16912294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:13 proc_lsst.multi:146 [INFO] found job 16912295 in provider astro proc_lsst.multi INFO: found job 16912295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:13 proc_lsst.multi:146 [INFO] found job 16912296 in provider astro proc_lsst.multi INFO: found job 16912296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:14 proc_lsst.multi:146 [INFO] found job 16912297 in provider astro proc_lsst.multi INFO: found job 16912297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:14 proc_lsst.multi:146 [INFO] found job 16912298 in provider astro proc_lsst.multi INFO: found job 16912298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:14 proc_lsst.multi:146 [INFO] found job 16912299 in provider astro proc_lsst.multi INFO: found job 16912299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:14 proc_lsst.multi:146 [INFO] found job 16912300 in provider astro proc_lsst.multi INFO: found job 16912300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:14 proc_lsst.multi:146 [INFO] found job 16912301 in provider astro proc_lsst.multi INFO: found job 16912301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:14 proc_lsst.multi:146 [INFO] found job 16912302 in provider astro proc_lsst.multi INFO: found job 16912302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:14 proc_lsst.multi:146 [INFO] found job 16912303 in provider astro proc_lsst.multi INFO: found job 16912303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:15 proc_lsst.multi:146 [INFO] found job 16912304 in provider astro proc_lsst.multi INFO: found job 16912304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:15 proc_lsst.multi:146 [INFO] found job 16912305 in provider astro proc_lsst.multi INFO: found job 16912305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:15 proc_lsst.multi:146 [INFO] found job 16912306 in provider astro proc_lsst.multi INFO: found job 16912306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:15 proc_lsst.multi:146 [INFO] found job 16912307 in provider astro proc_lsst.multi INFO: found job 16912307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:15 proc_lsst.multi:146 [INFO] found job 16912308 in provider astro proc_lsst.multi INFO: found job 16912308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:15 proc_lsst.multi:146 [INFO] found job 16912309 in provider astro proc_lsst.multi INFO: found job 16912309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:15 proc_lsst.multi:146 [INFO] found job 16912310 in provider astro proc_lsst.multi INFO: found job 16912310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:16 proc_lsst.multi:146 [INFO] found job 16912311 in provider astro proc_lsst.multi INFO: found job 16912311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:16 proc_lsst.multi:146 [INFO] found job 16912312 in provider astro proc_lsst.multi INFO: found job 16912312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:16 proc_lsst.multi:146 [INFO] found job 16912313 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912313 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:16 proc_lsst.multi:146 [INFO] found job 16912314 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912314 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:16 proc_lsst.multi:146 [INFO] found job 16912315 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912315 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:16 proc_lsst.multi:146 [INFO] found job 16912316 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912316 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:16 proc_lsst.multi:146 [INFO] found job 16912317 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:17 proc_lsst.multi:146 [INFO] found job 16912318 in provider escience_a40 proc_lsst.multi INFO: found job 16912318 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:17 proc_lsst.multi:146 [INFO] found job 16912319 in provider escience_a40 proc_lsst.multi INFO: found job 16912319 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:17 proc_lsst.multi:146 [INFO] found job 16912320 in provider escience_a40 proc_lsst.multi INFO: found job 16912320 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:17 proc_lsst.multi:146 [INFO] found job 16912321 in provider escience_a40 proc_lsst.multi INFO: found job 16912321 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:17 proc_lsst.multi:146 [INFO] found job 16912322 in provider escience_a40 proc_lsst.multi INFO: found job 16912322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:17 proc_lsst.multi:146 [INFO] found job 16912323 in provider escience_a40 proc_lsst.multi INFO: found job 16912323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:18 proc_lsst.multi:146 [INFO] found job 16912324 in provider escience_a40 proc_lsst.multi INFO: found job 16912324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:18 proc_lsst.multi:146 [INFO] found job 16912325 in provider escience_a40 proc_lsst.multi INFO: found job 16912325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:18 proc_lsst.multi:146 [INFO] found job 16912326 in provider escience_a40 proc_lsst.multi INFO: found job 16912326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:18 proc_lsst.multi:146 [INFO] found job 16912327 in provider escience_a40 proc_lsst.multi INFO: found job 16912327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:18 proc_lsst.multi:146 [INFO] found job 16912328 in provider escience_a40 proc_lsst.multi INFO: found job 16912328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:18 proc_lsst.multi:146 [INFO] found job 16912329 in provider escience_a40 proc_lsst.multi INFO: found job 16912329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:18 proc_lsst.multi:146 [INFO] found job 16912330 in provider escience_a40 proc_lsst.multi INFO: found job 16912330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:19 proc_lsst.multi:146 [INFO] found job 16912331 in provider escience_a40 proc_lsst.multi INFO: found job 16912331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:19 proc_lsst.multi:146 [INFO] found job 16912332 in provider escience_a40 proc_lsst.multi INFO: found job 16912332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:19 proc_lsst.multi:146 [INFO] found job 16912333 in provider escience_a40 proc_lsst.multi INFO: found job 16912333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:19 proc_lsst.multi:146 [INFO] found job 16912334 in provider escience_a40 proc_lsst.multi INFO: found job 16912334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:19 proc_lsst.multi:146 [INFO] found job 16912335 in provider escience_a40 proc_lsst.multi INFO: found job 16912335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:19 proc_lsst.multi:146 [INFO] found job 16912336 in provider escience_a40 proc_lsst.multi INFO: found job 16912336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:19 proc_lsst.multi:146 [INFO] found job 16912337 in provider escience_a40 proc_lsst.multi INFO: found job 16912337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:20 proc_lsst.multi:146 [INFO] found job 16912338 in provider checkpoint proc_lsst.multi INFO: found job 16912338 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:20 proc_lsst.multi:146 [INFO] found job 16912339 in provider checkpoint proc_lsst.multi INFO: found job 16912339 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:20 proc_lsst.multi:146 [INFO] found job 16912340 in provider checkpoint proc_lsst.multi INFO: found job 16912340 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:20 proc_lsst.multi:146 [INFO] found job 16912341 in provider checkpoint proc_lsst.multi INFO: found job 16912341 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:20 proc_lsst.multi:146 [INFO] found job 16912342 in provider checkpoint proc_lsst.multi INFO: found job 16912342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:20 proc_lsst.multi:146 [INFO] found job 16912343 in provider checkpoint proc_lsst.multi INFO: found job 16912343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:21 proc_lsst.multi:146 [INFO] found job 16912344 in provider checkpoint proc_lsst.multi INFO: found job 16912344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:21 proc_lsst.multi:146 [INFO] found job 16912345 in provider checkpoint proc_lsst.multi INFO: found job 16912345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:21 proc_lsst.multi:146 [INFO] found job 16912346 in provider checkpoint proc_lsst.multi INFO: found job 16912346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:21 proc_lsst.multi:146 [INFO] found job 16912347 in provider checkpoint proc_lsst.multi INFO: found job 16912347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:21 proc_lsst.multi:146 [INFO] found job 16912348 in provider checkpoint proc_lsst.multi INFO: found job 16912348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:21 proc_lsst.multi:146 [INFO] found job 16912349 in provider checkpoint proc_lsst.multi INFO: found job 16912349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:21 proc_lsst.multi:146 [INFO] found job 16912350 in provider checkpoint proc_lsst.multi INFO: found job 16912350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:22 proc_lsst.multi:146 [INFO] found job 16912351 in provider checkpoint proc_lsst.multi INFO: found job 16912351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:22 proc_lsst.multi:146 [INFO] found job 16912352 in provider checkpoint proc_lsst.multi INFO: found job 16912352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:22 proc_lsst.multi:146 [INFO] found job 16912353 in provider checkpoint proc_lsst.multi INFO: found job 16912353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:22 proc_lsst.multi:146 [INFO] found job 16912354 in provider checkpoint proc_lsst.multi INFO: found job 16912354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:22 proc_lsst.multi:146 [INFO] found job 16912355 in provider checkpoint proc_lsst.multi INFO: found job 16912355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:22 proc_lsst.multi:146 [INFO] found job 16912356 in provider checkpoint proc_lsst.multi INFO: found job 16912356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:22 proc_lsst.multi:146 [INFO] found job 16912359 in provider checkpoint proc_lsst.multi INFO: found job 16912359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:23 proc_lsst.multi:146 [INFO] found job 16912360 in provider checkpoint proc_lsst.multi INFO: found job 16912360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:23 proc_lsst.multi:146 [INFO] found job 16912361 in provider checkpoint proc_lsst.multi INFO: found job 16912361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:23 proc_lsst.multi:146 [INFO] found job 16912362 in provider checkpoint proc_lsst.multi INFO: found job 16912362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:23 proc_lsst.multi:146 [INFO] found job 16912363 in provider checkpoint proc_lsst.multi INFO: found job 16912363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:23 proc_lsst.multi:146 [INFO] found job 16912364 in provider checkpoint proc_lsst.multi INFO: found job 16912364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:23 proc_lsst.multi:146 [INFO] found job 16912365 in provider checkpoint proc_lsst.multi INFO: found job 16912365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:23 proc_lsst.multi:146 [INFO] found job 16912366 in provider checkpoint proc_lsst.multi INFO: found job 16912366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:26 proc_lsst.multi:146 [INFO] found job 16912367 in provider checkpoint proc_lsst.multi INFO: found job 16912367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:26 proc_lsst.multi:146 [INFO] found job 16912368 in provider checkpoint proc_lsst.multi INFO: found job 16912368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:26 proc_lsst.multi:146 [INFO] found job 16912369 in provider checkpoint proc_lsst.multi INFO: found job 16912369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:26 proc_lsst.multi:146 [INFO] found job 16912370 in provider checkpoint proc_lsst.multi INFO: found job 16912370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:28 proc_lsst.multi:146 [INFO] found job 16912371 in provider checkpoint proc_lsst.multi INFO: found job 16912371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:28 proc_lsst.multi:146 [INFO] found job 16912372 in provider checkpoint proc_lsst.multi INFO: found job 16912372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:29 proc_lsst.multi:146 [INFO] found job 16912373 in provider checkpoint proc_lsst.multi INFO: found job 16912373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:29 proc_lsst.multi:146 [INFO] found job 16912374 in provider checkpoint proc_lsst.multi INFO: found job 16912374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:30 proc_lsst.multi:146 [INFO] found job 16912375 in provider checkpoint proc_lsst.multi INFO: found job 16912375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:31 proc_lsst.multi:146 [INFO] found job 16912378 in provider checkpoint proc_lsst.multi INFO: found job 16912378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.dflow INFO: Task 1040 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64945d563079654efc7edf65a5ed36e5 with result from task 1040 parsl.dataflow.dflow DEBUG: Task 1040 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/9b04132a-2510-4c21-ba68-f822b265e350_calibrate_855138_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855138/9b04132a-2510-4c21-ba68-f822b265e350_calibrate_855138_61.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 2024-03-08 07:24:46 proc_lsst.multi:146 [INFO] found job 75775 in provider local proc_lsst.multi INFO: found job 75775 in provider local 2024-03-08 07:24:46 proc_lsst.multi:146 [INFO] found job 16912283 in provider astro proc_lsst.multi INFO: found job 16912283 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:46 proc_lsst.multi:146 [INFO] found job 16912284 in provider astro proc_lsst.multi INFO: found job 16912284 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:46 proc_lsst.multi:146 [INFO] found job 16912285 in provider astro proc_lsst.multi INFO: found job 16912285 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:46 proc_lsst.multi:146 [INFO] found job 16912286 in provider astro proc_lsst.multi INFO: found job 16912286 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:46 proc_lsst.multi:146 [INFO] found job 16912287 in provider astro proc_lsst.multi INFO: found job 16912287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:47 proc_lsst.multi:146 [INFO] found job 16912288 in provider astro proc_lsst.multi INFO: found job 16912288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:47 proc_lsst.multi:146 [INFO] found job 16912289 in provider astro proc_lsst.multi INFO: found job 16912289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:47 proc_lsst.multi:146 [INFO] found job 16912290 in provider astro proc_lsst.multi INFO: found job 16912290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:47 proc_lsst.multi:146 [INFO] found job 16912291 in provider astro proc_lsst.multi INFO: found job 16912291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:47 proc_lsst.multi:146 [INFO] found job 16912292 in provider astro proc_lsst.multi INFO: found job 16912292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:47 proc_lsst.multi:146 [INFO] found job 16912293 in provider astro proc_lsst.multi INFO: found job 16912293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:47 proc_lsst.multi:146 [INFO] found job 16912294 in provider astro proc_lsst.multi INFO: found job 16912294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:48 proc_lsst.multi:146 [INFO] found job 16912295 in provider astro proc_lsst.multi INFO: found job 16912295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:48 proc_lsst.multi:146 [INFO] found job 16912296 in provider astro proc_lsst.multi INFO: found job 16912296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:48 proc_lsst.multi:146 [INFO] found job 16912297 in provider astro proc_lsst.multi INFO: found job 16912297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:48 proc_lsst.multi:146 [INFO] found job 16912298 in provider astro proc_lsst.multi INFO: found job 16912298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:48 proc_lsst.multi:146 [INFO] found job 16912299 in provider astro proc_lsst.multi INFO: found job 16912299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:48 proc_lsst.multi:146 [INFO] found job 16912300 in provider astro proc_lsst.multi INFO: found job 16912300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:48 proc_lsst.multi:146 [INFO] found job 16912301 in provider astro proc_lsst.multi INFO: found job 16912301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:49 proc_lsst.multi:146 [INFO] found job 16912302 in provider astro proc_lsst.multi INFO: found job 16912302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:49 proc_lsst.multi:146 [INFO] found job 16912303 in provider astro proc_lsst.multi INFO: found job 16912303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:49 proc_lsst.multi:146 [INFO] found job 16912304 in provider astro proc_lsst.multi INFO: found job 16912304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:49 proc_lsst.multi:146 [INFO] found job 16912305 in provider astro proc_lsst.multi INFO: found job 16912305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:49 proc_lsst.multi:146 [INFO] found job 16912306 in provider astro proc_lsst.multi INFO: found job 16912306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:49 proc_lsst.multi:146 [INFO] found job 16912307 in provider astro proc_lsst.multi INFO: found job 16912307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:49 proc_lsst.multi:146 [INFO] found job 16912308 in provider astro proc_lsst.multi INFO: found job 16912308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:50 proc_lsst.multi:146 [INFO] found job 16912309 in provider astro proc_lsst.multi INFO: found job 16912309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:50 proc_lsst.multi:146 [INFO] found job 16912310 in provider astro proc_lsst.multi INFO: found job 16912310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:50 proc_lsst.multi:146 [INFO] found job 16912311 in provider astro proc_lsst.multi INFO: found job 16912311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:50 proc_lsst.multi:146 [INFO] found job 16912312 in provider astro proc_lsst.multi INFO: found job 16912312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:50 proc_lsst.multi:146 [INFO] found job 16912313 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912313 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:50 proc_lsst.multi:146 [INFO] found job 16912314 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912314 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:50 proc_lsst.multi:146 [INFO] found job 16912315 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912315 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:51 proc_lsst.multi:146 [INFO] found job 16912316 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912316 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:51 proc_lsst.multi:146 [INFO] found job 16912317 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:51 proc_lsst.multi:146 [INFO] found job 16912318 in provider escience_a40 proc_lsst.multi INFO: found job 16912318 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:51 proc_lsst.multi:146 [INFO] found job 16912319 in provider escience_a40 proc_lsst.multi INFO: found job 16912319 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:51 proc_lsst.multi:146 [INFO] found job 16912320 in provider escience_a40 proc_lsst.multi INFO: found job 16912320 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:51 proc_lsst.multi:146 [INFO] found job 16912321 in provider escience_a40 proc_lsst.multi INFO: found job 16912321 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:51 proc_lsst.multi:146 [INFO] found job 16912322 in provider escience_a40 proc_lsst.multi INFO: found job 16912322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:52 proc_lsst.multi:146 [INFO] found job 16912323 in provider escience_a40 proc_lsst.multi INFO: found job 16912323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:52 proc_lsst.multi:146 [INFO] found job 16912324 in provider escience_a40 proc_lsst.multi INFO: found job 16912324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:52 proc_lsst.multi:146 [INFO] found job 16912325 in provider escience_a40 proc_lsst.multi INFO: found job 16912325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:52 proc_lsst.multi:146 [INFO] found job 16912326 in provider escience_a40 proc_lsst.multi INFO: found job 16912326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:52 proc_lsst.multi:146 [INFO] found job 16912327 in provider escience_a40 proc_lsst.multi INFO: found job 16912327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:52 proc_lsst.multi:146 [INFO] found job 16912328 in provider escience_a40 proc_lsst.multi INFO: found job 16912328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:53 proc_lsst.multi:146 [INFO] found job 16912329 in provider escience_a40 proc_lsst.multi INFO: found job 16912329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:53 proc_lsst.multi:146 [INFO] found job 16912330 in provider escience_a40 proc_lsst.multi INFO: found job 16912330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:53 proc_lsst.multi:146 [INFO] found job 16912331 in provider escience_a40 proc_lsst.multi INFO: found job 16912331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:53 proc_lsst.multi:146 [INFO] found job 16912332 in provider escience_a40 proc_lsst.multi INFO: found job 16912332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:53 proc_lsst.multi:146 [INFO] found job 16912333 in provider escience_a40 proc_lsst.multi INFO: found job 16912333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:53 proc_lsst.multi:146 [INFO] found job 16912334 in provider escience_a40 proc_lsst.multi INFO: found job 16912334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:53 proc_lsst.multi:146 [INFO] found job 16912335 in provider escience_a40 proc_lsst.multi INFO: found job 16912335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:54 proc_lsst.multi:146 [INFO] found job 16912336 in provider escience_a40 proc_lsst.multi INFO: found job 16912336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:54 proc_lsst.multi:146 [INFO] found job 16912337 in provider escience_a40 proc_lsst.multi INFO: found job 16912337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:54 proc_lsst.multi:146 [INFO] found job 16912338 in provider checkpoint proc_lsst.multi INFO: found job 16912338 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:54 proc_lsst.multi:146 [INFO] found job 16912339 in provider checkpoint proc_lsst.multi INFO: found job 16912339 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:54 proc_lsst.multi:146 [INFO] found job 16912340 in provider checkpoint proc_lsst.multi INFO: found job 16912340 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:54 proc_lsst.multi:146 [INFO] found job 16912341 in provider checkpoint proc_lsst.multi INFO: found job 16912341 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:54 proc_lsst.multi:146 [INFO] found job 16912342 in provider checkpoint proc_lsst.multi INFO: found job 16912342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:55 proc_lsst.multi:146 [INFO] found job 16912343 in provider checkpoint proc_lsst.multi INFO: found job 16912343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:55 proc_lsst.multi:146 [INFO] found job 16912344 in provider checkpoint proc_lsst.multi INFO: found job 16912344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:55 proc_lsst.multi:146 [INFO] found job 16912345 in provider checkpoint proc_lsst.multi INFO: found job 16912345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:55 proc_lsst.multi:146 [INFO] found job 16912346 in provider checkpoint proc_lsst.multi INFO: found job 16912346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:55 proc_lsst.multi:146 [INFO] found job 16912347 in provider checkpoint proc_lsst.multi INFO: found job 16912347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:55 proc_lsst.multi:146 [INFO] found job 16912348 in provider checkpoint proc_lsst.multi INFO: found job 16912348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:55 proc_lsst.multi:146 [INFO] found job 16912349 in provider checkpoint proc_lsst.multi INFO: found job 16912349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:56 proc_lsst.multi:146 [INFO] found job 16912350 in provider checkpoint proc_lsst.multi INFO: found job 16912350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:56 proc_lsst.multi:146 [INFO] found job 16912351 in provider checkpoint proc_lsst.multi INFO: found job 16912351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:56 proc_lsst.multi:146 [INFO] found job 16912352 in provider checkpoint proc_lsst.multi INFO: found job 16912352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:56 proc_lsst.multi:146 [INFO] found job 16912353 in provider checkpoint proc_lsst.multi INFO: found job 16912353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:56 proc_lsst.multi:146 [INFO] found job 16912354 in provider checkpoint proc_lsst.multi INFO: found job 16912354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:56 proc_lsst.multi:146 [INFO] found job 16912355 in provider checkpoint proc_lsst.multi INFO: found job 16912355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:56 proc_lsst.multi:146 [INFO] found job 16912356 in provider checkpoint proc_lsst.multi INFO: found job 16912356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:57 proc_lsst.multi:146 [INFO] found job 16912359 in provider checkpoint proc_lsst.multi INFO: found job 16912359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:57 proc_lsst.multi:146 [INFO] found job 16912360 in provider checkpoint proc_lsst.multi INFO: found job 16912360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:57 proc_lsst.multi:146 [INFO] found job 16912361 in provider checkpoint proc_lsst.multi INFO: found job 16912361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:57 proc_lsst.multi:146 [INFO] found job 16912362 in provider checkpoint proc_lsst.multi INFO: found job 16912362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:57 proc_lsst.multi:146 [INFO] found job 16912363 in provider checkpoint proc_lsst.multi INFO: found job 16912363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:58 proc_lsst.multi:146 [INFO] found job 16912364 in provider checkpoint proc_lsst.multi INFO: found job 16912364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:58 proc_lsst.multi:146 [INFO] found job 16912365 in provider checkpoint proc_lsst.multi INFO: found job 16912365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:58 proc_lsst.multi:146 [INFO] found job 16912366 in provider checkpoint proc_lsst.multi INFO: found job 16912366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:58 proc_lsst.multi:146 [INFO] found job 16912367 in provider checkpoint proc_lsst.multi INFO: found job 16912367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:58 proc_lsst.multi:146 [INFO] found job 16912368 in provider checkpoint proc_lsst.multi INFO: found job 16912368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:58 proc_lsst.multi:146 [INFO] found job 16912369 in provider checkpoint proc_lsst.multi INFO: found job 16912369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:58 proc_lsst.multi:146 [INFO] found job 16912370 in provider checkpoint proc_lsst.multi INFO: found job 16912370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:59 proc_lsst.multi:146 [INFO] found job 16912371 in provider checkpoint proc_lsst.multi INFO: found job 16912371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:59 proc_lsst.multi:146 [INFO] found job 16912372 in provider checkpoint proc_lsst.multi INFO: found job 16912372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:59 proc_lsst.multi:146 [INFO] found job 16912373 in provider checkpoint proc_lsst.multi INFO: found job 16912373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:59 proc_lsst.multi:146 [INFO] found job 16912374 in provider checkpoint proc_lsst.multi INFO: found job 16912374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:59 proc_lsst.multi:146 [INFO] found job 16912375 in provider checkpoint proc_lsst.multi INFO: found job 16912375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:24:59 proc_lsst.multi:146 [INFO] found job 16912378 in provider checkpoint proc_lsst.multi INFO: found job 16912378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 2024-03-08 07:25:19 proc_lsst.multi:146 [INFO] found job 75775 in provider local proc_lsst.multi INFO: found job 75775 in provider local 2024-03-08 07:25:20 proc_lsst.multi:146 [INFO] found job 16912283 in provider astro proc_lsst.multi INFO: found job 16912283 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:20 proc_lsst.multi:146 [INFO] found job 16912284 in provider astro proc_lsst.multi INFO: found job 16912284 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:20 proc_lsst.multi:146 [INFO] found job 16912285 in provider astro proc_lsst.multi INFO: found job 16912285 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:20 proc_lsst.multi:146 [INFO] found job 16912286 in provider astro proc_lsst.multi INFO: found job 16912286 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:20 proc_lsst.multi:146 [INFO] found job 16912287 in provider astro proc_lsst.multi INFO: found job 16912287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:20 proc_lsst.multi:146 [INFO] found job 16912288 in provider astro proc_lsst.multi INFO: found job 16912288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:20 proc_lsst.multi:146 [INFO] found job 16912289 in provider astro proc_lsst.multi INFO: found job 16912289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:21 proc_lsst.multi:146 [INFO] found job 16912290 in provider astro proc_lsst.multi INFO: found job 16912290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:21 proc_lsst.multi:146 [INFO] found job 16912291 in provider astro proc_lsst.multi INFO: found job 16912291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:21 proc_lsst.multi:146 [INFO] found job 16912292 in provider astro proc_lsst.multi INFO: found job 16912292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:21 proc_lsst.multi:146 [INFO] found job 16912293 in provider astro proc_lsst.multi INFO: found job 16912293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:21 proc_lsst.multi:146 [INFO] found job 16912294 in provider astro proc_lsst.multi INFO: found job 16912294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:21 proc_lsst.multi:146 [INFO] found job 16912295 in provider astro proc_lsst.multi INFO: found job 16912295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:22 proc_lsst.multi:146 [INFO] found job 16912296 in provider astro proc_lsst.multi INFO: found job 16912296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:22 proc_lsst.multi:146 [INFO] found job 16912297 in provider astro proc_lsst.multi INFO: found job 16912297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:22 proc_lsst.multi:146 [INFO] found job 16912298 in provider astro proc_lsst.multi INFO: found job 16912298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:22 proc_lsst.multi:146 [INFO] found job 16912299 in provider astro proc_lsst.multi INFO: found job 16912299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:22 proc_lsst.multi:146 [INFO] found job 16912300 in provider astro proc_lsst.multi INFO: found job 16912300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:22 proc_lsst.multi:146 [INFO] found job 16912301 in provider astro proc_lsst.multi INFO: found job 16912301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:22 proc_lsst.multi:146 [INFO] found job 16912302 in provider astro proc_lsst.multi INFO: found job 16912302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:23 proc_lsst.multi:146 [INFO] found job 16912303 in provider astro proc_lsst.multi INFO: found job 16912303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:23 proc_lsst.multi:146 [INFO] found job 16912304 in provider astro proc_lsst.multi INFO: found job 16912304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:23 proc_lsst.multi:146 [INFO] found job 16912305 in provider astro proc_lsst.multi INFO: found job 16912305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:23 proc_lsst.multi:146 [INFO] found job 16912306 in provider astro proc_lsst.multi INFO: found job 16912306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:23 proc_lsst.multi:146 [INFO] found job 16912307 in provider astro proc_lsst.multi INFO: found job 16912307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:23 proc_lsst.multi:146 [INFO] found job 16912308 in provider astro proc_lsst.multi INFO: found job 16912308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:23 proc_lsst.multi:146 [INFO] found job 16912309 in provider astro proc_lsst.multi INFO: found job 16912309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:24 proc_lsst.multi:146 [INFO] found job 16912310 in provider astro proc_lsst.multi INFO: found job 16912310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:24 proc_lsst.multi:146 [INFO] found job 16912311 in provider astro proc_lsst.multi INFO: found job 16912311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:24 proc_lsst.multi:146 [INFO] found job 16912312 in provider astro proc_lsst.multi INFO: found job 16912312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:24 proc_lsst.multi:146 [INFO] found job 16912313 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912313 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:24 proc_lsst.multi:146 [INFO] found job 16912314 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912314 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:24 proc_lsst.multi:146 [INFO] found job 16912315 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912315 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:24 proc_lsst.multi:146 [INFO] found job 16912316 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912316 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:25 proc_lsst.multi:146 [INFO] found job 16912317 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:25 proc_lsst.multi:146 [INFO] found job 16912318 in provider escience_a40 proc_lsst.multi INFO: found job 16912318 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:25 proc_lsst.multi:146 [INFO] found job 16912319 in provider escience_a40 proc_lsst.multi INFO: found job 16912319 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:25 proc_lsst.multi:146 [INFO] found job 16912320 in provider escience_a40 proc_lsst.multi INFO: found job 16912320 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:25 proc_lsst.multi:146 [INFO] found job 16912321 in provider escience_a40 proc_lsst.multi INFO: found job 16912321 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:25 proc_lsst.multi:146 [INFO] found job 16912322 in provider escience_a40 proc_lsst.multi INFO: found job 16912322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:26 proc_lsst.multi:146 [INFO] found job 16912323 in provider escience_a40 proc_lsst.multi INFO: found job 16912323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:26 proc_lsst.multi:146 [INFO] found job 16912324 in provider escience_a40 proc_lsst.multi INFO: found job 16912324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:26 proc_lsst.multi:146 [INFO] found job 16912325 in provider escience_a40 proc_lsst.multi INFO: found job 16912325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:26 proc_lsst.multi:146 [INFO] found job 16912326 in provider escience_a40 proc_lsst.multi INFO: found job 16912326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:26 proc_lsst.multi:146 [INFO] found job 16912327 in provider escience_a40 proc_lsst.multi INFO: found job 16912327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:26 proc_lsst.multi:146 [INFO] found job 16912328 in provider escience_a40 proc_lsst.multi INFO: found job 16912328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:26 proc_lsst.multi:146 [INFO] found job 16912329 in provider escience_a40 proc_lsst.multi INFO: found job 16912329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:27 proc_lsst.multi:146 [INFO] found job 16912330 in provider escience_a40 proc_lsst.multi INFO: found job 16912330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:27 proc_lsst.multi:146 [INFO] found job 16912331 in provider escience_a40 proc_lsst.multi INFO: found job 16912331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:27 proc_lsst.multi:146 [INFO] found job 16912332 in provider escience_a40 proc_lsst.multi INFO: found job 16912332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:27 proc_lsst.multi:146 [INFO] found job 16912333 in provider escience_a40 proc_lsst.multi INFO: found job 16912333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:27 proc_lsst.multi:146 [INFO] found job 16912334 in provider escience_a40 proc_lsst.multi INFO: found job 16912334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:27 proc_lsst.multi:146 [INFO] found job 16912335 in provider escience_a40 proc_lsst.multi INFO: found job 16912335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:28 proc_lsst.multi:146 [INFO] found job 16912336 in provider escience_a40 proc_lsst.multi INFO: found job 16912336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:28 proc_lsst.multi:146 [INFO] found job 16912337 in provider escience_a40 proc_lsst.multi INFO: found job 16912337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:28 proc_lsst.multi:146 [INFO] found job 16912338 in provider checkpoint proc_lsst.multi INFO: found job 16912338 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:28 proc_lsst.multi:146 [INFO] found job 16912339 in provider checkpoint proc_lsst.multi INFO: found job 16912339 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:28 proc_lsst.multi:146 [INFO] found job 16912340 in provider checkpoint proc_lsst.multi INFO: found job 16912340 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:28 proc_lsst.multi:146 [INFO] found job 16912341 in provider checkpoint proc_lsst.multi INFO: found job 16912341 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:28 proc_lsst.multi:146 [INFO] found job 16912342 in provider checkpoint proc_lsst.multi INFO: found job 16912342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:29 proc_lsst.multi:146 [INFO] found job 16912343 in provider checkpoint proc_lsst.multi INFO: found job 16912343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:29 proc_lsst.multi:146 [INFO] found job 16912344 in provider checkpoint proc_lsst.multi INFO: found job 16912344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:29 proc_lsst.multi:146 [INFO] found job 16912345 in provider checkpoint proc_lsst.multi INFO: found job 16912345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:29 proc_lsst.multi:146 [INFO] found job 16912346 in provider checkpoint proc_lsst.multi INFO: found job 16912346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:29 proc_lsst.multi:146 [INFO] found job 16912347 in provider checkpoint proc_lsst.multi INFO: found job 16912347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:29 proc_lsst.multi:146 [INFO] found job 16912348 in provider checkpoint proc_lsst.multi INFO: found job 16912348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:30 proc_lsst.multi:146 [INFO] found job 16912349 in provider checkpoint proc_lsst.multi INFO: found job 16912349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:30 proc_lsst.multi:146 [INFO] found job 16912350 in provider checkpoint proc_lsst.multi INFO: found job 16912350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:30 proc_lsst.multi:146 [INFO] found job 16912351 in provider checkpoint proc_lsst.multi INFO: found job 16912351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:30 proc_lsst.multi:146 [INFO] found job 16912352 in provider checkpoint proc_lsst.multi INFO: found job 16912352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:30 proc_lsst.multi:146 [INFO] found job 16912353 in provider checkpoint proc_lsst.multi INFO: found job 16912353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:30 proc_lsst.multi:146 [INFO] found job 16912354 in provider checkpoint proc_lsst.multi INFO: found job 16912354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:30 proc_lsst.multi:146 [INFO] found job 16912355 in provider checkpoint proc_lsst.multi INFO: found job 16912355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:31 proc_lsst.multi:146 [INFO] found job 16912356 in provider checkpoint proc_lsst.multi INFO: found job 16912356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:31 proc_lsst.multi:146 [INFO] found job 16912359 in provider checkpoint proc_lsst.multi INFO: found job 16912359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:31 proc_lsst.multi:146 [INFO] found job 16912360 in provider checkpoint proc_lsst.multi INFO: found job 16912360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:31 proc_lsst.multi:146 [INFO] found job 16912361 in provider checkpoint proc_lsst.multi INFO: found job 16912361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:31 proc_lsst.multi:146 [INFO] found job 16912362 in provider checkpoint proc_lsst.multi INFO: found job 16912362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:31 proc_lsst.multi:146 [INFO] found job 16912363 in provider checkpoint proc_lsst.multi INFO: found job 16912363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:31 proc_lsst.multi:146 [INFO] found job 16912364 in provider checkpoint proc_lsst.multi INFO: found job 16912364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:32 proc_lsst.multi:146 [INFO] found job 16912365 in provider checkpoint proc_lsst.multi INFO: found job 16912365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:32 proc_lsst.multi:146 [INFO] found job 16912366 in provider checkpoint proc_lsst.multi INFO: found job 16912366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:32 proc_lsst.multi:146 [INFO] found job 16912367 in provider checkpoint proc_lsst.multi INFO: found job 16912367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:32 proc_lsst.multi:146 [INFO] found job 16912368 in provider checkpoint proc_lsst.multi INFO: found job 16912368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:32 proc_lsst.multi:146 [INFO] found job 16912369 in provider checkpoint proc_lsst.multi INFO: found job 16912369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:32 proc_lsst.multi:146 [INFO] found job 16912370 in provider checkpoint proc_lsst.multi INFO: found job 16912370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:33 proc_lsst.multi:146 [INFO] found job 16912371 in provider checkpoint proc_lsst.multi INFO: found job 16912371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:33 proc_lsst.multi:146 [INFO] found job 16912372 in provider checkpoint proc_lsst.multi INFO: found job 16912372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:33 proc_lsst.multi:146 [INFO] found job 16912373 in provider checkpoint proc_lsst.multi INFO: found job 16912373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:33 proc_lsst.multi:146 [INFO] found job 16912374 in provider checkpoint proc_lsst.multi INFO: found job 16912374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:33 proc_lsst.multi:146 [INFO] found job 16912375 in provider checkpoint proc_lsst.multi INFO: found job 16912375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:33 proc_lsst.multi:146 [INFO] found job 16912378 in provider checkpoint proc_lsst.multi INFO: found job 16912378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 2024-03-08 07:25:53 proc_lsst.multi:146 [INFO] found job 75775 in provider local proc_lsst.multi INFO: found job 75775 in provider local 2024-03-08 07:25:54 proc_lsst.multi:146 [INFO] found job 16912283 in provider astro proc_lsst.multi INFO: found job 16912283 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:54 proc_lsst.multi:146 [INFO] found job 16912284 in provider astro proc_lsst.multi INFO: found job 16912284 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:54 proc_lsst.multi:146 [INFO] found job 16912285 in provider astro proc_lsst.multi INFO: found job 16912285 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:54 proc_lsst.multi:146 [INFO] found job 16912286 in provider astro proc_lsst.multi INFO: found job 16912286 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:54 proc_lsst.multi:146 [INFO] found job 16912287 in provider astro proc_lsst.multi INFO: found job 16912287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:54 proc_lsst.multi:146 [INFO] found job 16912288 in provider astro proc_lsst.multi INFO: found job 16912288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:54 proc_lsst.multi:146 [INFO] found job 16912289 in provider astro proc_lsst.multi INFO: found job 16912289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:55 proc_lsst.multi:146 [INFO] found job 16912290 in provider astro proc_lsst.multi INFO: found job 16912290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:55 proc_lsst.multi:146 [INFO] found job 16912291 in provider astro proc_lsst.multi INFO: found job 16912291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:55 proc_lsst.multi:146 [INFO] found job 16912292 in provider astro proc_lsst.multi INFO: found job 16912292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:55 proc_lsst.multi:146 [INFO] found job 16912293 in provider astro proc_lsst.multi INFO: found job 16912293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:55 proc_lsst.multi:146 [INFO] found job 16912294 in provider astro proc_lsst.multi INFO: found job 16912294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:55 proc_lsst.multi:146 [INFO] found job 16912295 in provider astro proc_lsst.multi INFO: found job 16912295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:55 proc_lsst.multi:146 [INFO] found job 16912296 in provider astro proc_lsst.multi INFO: found job 16912296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:56 proc_lsst.multi:146 [INFO] found job 16912297 in provider astro proc_lsst.multi INFO: found job 16912297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:56 proc_lsst.multi:146 [INFO] found job 16912298 in provider astro proc_lsst.multi INFO: found job 16912298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:56 proc_lsst.multi:146 [INFO] found job 16912299 in provider astro proc_lsst.multi INFO: found job 16912299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:56 proc_lsst.multi:146 [INFO] found job 16912300 in provider astro proc_lsst.multi INFO: found job 16912300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:56 proc_lsst.multi:146 [INFO] found job 16912301 in provider astro proc_lsst.multi INFO: found job 16912301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:56 proc_lsst.multi:146 [INFO] found job 16912302 in provider astro proc_lsst.multi INFO: found job 16912302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:56 proc_lsst.multi:146 [INFO] found job 16912303 in provider astro proc_lsst.multi INFO: found job 16912303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:57 proc_lsst.multi:146 [INFO] found job 16912304 in provider astro proc_lsst.multi INFO: found job 16912304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:57 proc_lsst.multi:146 [INFO] found job 16912305 in provider astro proc_lsst.multi INFO: found job 16912305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:57 proc_lsst.multi:146 [INFO] found job 16912306 in provider astro proc_lsst.multi INFO: found job 16912306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:57 proc_lsst.multi:146 [INFO] found job 16912307 in provider astro proc_lsst.multi INFO: found job 16912307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:57 proc_lsst.multi:146 [INFO] found job 16912308 in provider astro proc_lsst.multi INFO: found job 16912308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:57 proc_lsst.multi:146 [INFO] found job 16912309 in provider astro proc_lsst.multi INFO: found job 16912309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:57 proc_lsst.multi:146 [INFO] found job 16912310 in provider astro proc_lsst.multi INFO: found job 16912310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:58 proc_lsst.multi:146 [INFO] found job 16912311 in provider astro proc_lsst.multi INFO: found job 16912311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:58 proc_lsst.multi:146 [INFO] found job 16912312 in provider astro proc_lsst.multi INFO: found job 16912312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:58 proc_lsst.multi:146 [INFO] found job 16912313 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912313 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:58 proc_lsst.multi:146 [INFO] found job 16912314 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912314 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:58 proc_lsst.multi:146 [INFO] found job 16912315 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912315 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:59 proc_lsst.multi:146 [INFO] found job 16912316 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912316 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:59 proc_lsst.multi:146 [INFO] found job 16912317 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:59 proc_lsst.multi:146 [INFO] found job 16912318 in provider escience_a40 proc_lsst.multi INFO: found job 16912318 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:59 proc_lsst.multi:146 [INFO] found job 16912319 in provider escience_a40 proc_lsst.multi INFO: found job 16912319 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:59 proc_lsst.multi:146 [INFO] found job 16912320 in provider escience_a40 proc_lsst.multi INFO: found job 16912320 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:25:59 proc_lsst.multi:146 [INFO] found job 16912321 in provider escience_a40 proc_lsst.multi INFO: found job 16912321 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:00 proc_lsst.multi:146 [INFO] found job 16912322 in provider escience_a40 proc_lsst.multi INFO: found job 16912322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:00 proc_lsst.multi:146 [INFO] found job 16912323 in provider escience_a40 proc_lsst.multi INFO: found job 16912323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:00 proc_lsst.multi:146 [INFO] found job 16912324 in provider escience_a40 proc_lsst.multi INFO: found job 16912324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:00 proc_lsst.multi:146 [INFO] found job 16912325 in provider escience_a40 proc_lsst.multi INFO: found job 16912325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:00 proc_lsst.multi:146 [INFO] found job 16912326 in provider escience_a40 proc_lsst.multi INFO: found job 16912326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:00 proc_lsst.multi:146 [INFO] found job 16912327 in provider escience_a40 proc_lsst.multi INFO: found job 16912327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:01 proc_lsst.multi:146 [INFO] found job 16912328 in provider escience_a40 proc_lsst.multi INFO: found job 16912328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:01 proc_lsst.multi:146 [INFO] found job 16912329 in provider escience_a40 proc_lsst.multi INFO: found job 16912329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:01 proc_lsst.multi:146 [INFO] found job 16912330 in provider escience_a40 proc_lsst.multi INFO: found job 16912330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:01 proc_lsst.multi:146 [INFO] found job 16912331 in provider escience_a40 proc_lsst.multi INFO: found job 16912331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:01 proc_lsst.multi:146 [INFO] found job 16912332 in provider escience_a40 proc_lsst.multi INFO: found job 16912332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:01 proc_lsst.multi:146 [INFO] found job 16912333 in provider escience_a40 proc_lsst.multi INFO: found job 16912333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:02 proc_lsst.multi:146 [INFO] found job 16912334 in provider escience_a40 proc_lsst.multi INFO: found job 16912334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:02 proc_lsst.multi:146 [INFO] found job 16912335 in provider escience_a40 proc_lsst.multi INFO: found job 16912335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:02 proc_lsst.multi:146 [INFO] found job 16912336 in provider escience_a40 proc_lsst.multi INFO: found job 16912336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:02 proc_lsst.multi:146 [INFO] found job 16912337 in provider escience_a40 proc_lsst.multi INFO: found job 16912337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:02 proc_lsst.multi:146 [INFO] found job 16912338 in provider checkpoint proc_lsst.multi INFO: found job 16912338 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:02 proc_lsst.multi:146 [INFO] found job 16912339 in provider checkpoint proc_lsst.multi INFO: found job 16912339 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:02 proc_lsst.multi:146 [INFO] found job 16912340 in provider checkpoint proc_lsst.multi INFO: found job 16912340 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:03 proc_lsst.multi:146 [INFO] found job 16912341 in provider checkpoint proc_lsst.multi INFO: found job 16912341 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:03 proc_lsst.multi:146 [INFO] found job 16912342 in provider checkpoint proc_lsst.multi INFO: found job 16912342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:03 proc_lsst.multi:146 [INFO] found job 16912343 in provider checkpoint proc_lsst.multi INFO: found job 16912343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:03 proc_lsst.multi:146 [INFO] found job 16912344 in provider checkpoint proc_lsst.multi INFO: found job 16912344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:03 proc_lsst.multi:146 [INFO] found job 16912345 in provider checkpoint proc_lsst.multi INFO: found job 16912345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:03 proc_lsst.multi:146 [INFO] found job 16912346 in provider checkpoint proc_lsst.multi INFO: found job 16912346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:04 proc_lsst.multi:146 [INFO] found job 16912347 in provider checkpoint proc_lsst.multi INFO: found job 16912347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:04 proc_lsst.multi:146 [INFO] found job 16912348 in provider checkpoint proc_lsst.multi INFO: found job 16912348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:04 proc_lsst.multi:146 [INFO] found job 16912349 in provider checkpoint proc_lsst.multi INFO: found job 16912349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:04 proc_lsst.multi:146 [INFO] found job 16912350 in provider checkpoint proc_lsst.multi INFO: found job 16912350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:04 proc_lsst.multi:146 [INFO] found job 16912351 in provider checkpoint proc_lsst.multi INFO: found job 16912351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:04 proc_lsst.multi:146 [INFO] found job 16912352 in provider checkpoint proc_lsst.multi INFO: found job 16912352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:05 proc_lsst.multi:146 [INFO] found job 16912353 in provider checkpoint proc_lsst.multi INFO: found job 16912353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:05 proc_lsst.multi:146 [INFO] found job 16912354 in provider checkpoint proc_lsst.multi INFO: found job 16912354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:05 proc_lsst.multi:146 [INFO] found job 16912355 in provider checkpoint proc_lsst.multi INFO: found job 16912355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:05 proc_lsst.multi:146 [INFO] found job 16912356 in provider checkpoint proc_lsst.multi INFO: found job 16912356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:05 proc_lsst.multi:146 [INFO] found job 16912359 in provider checkpoint proc_lsst.multi INFO: found job 16912359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:05 proc_lsst.multi:146 [INFO] found job 16912360 in provider checkpoint proc_lsst.multi INFO: found job 16912360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:05 proc_lsst.multi:146 [INFO] found job 16912361 in provider checkpoint proc_lsst.multi INFO: found job 16912361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:06 proc_lsst.multi:146 [INFO] found job 16912362 in provider checkpoint proc_lsst.multi INFO: found job 16912362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:06 proc_lsst.multi:146 [INFO] found job 16912363 in provider checkpoint proc_lsst.multi INFO: found job 16912363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:06 proc_lsst.multi:146 [INFO] found job 16912364 in provider checkpoint proc_lsst.multi INFO: found job 16912364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:06 proc_lsst.multi:146 [INFO] found job 16912365 in provider checkpoint proc_lsst.multi INFO: found job 16912365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:06 proc_lsst.multi:146 [INFO] found job 16912366 in provider checkpoint proc_lsst.multi INFO: found job 16912366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:06 proc_lsst.multi:146 [INFO] found job 16912367 in provider checkpoint proc_lsst.multi INFO: found job 16912367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:06 proc_lsst.multi:146 [INFO] found job 16912368 in provider checkpoint proc_lsst.multi INFO: found job 16912368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:07 proc_lsst.multi:146 [INFO] found job 16912369 in provider checkpoint proc_lsst.multi INFO: found job 16912369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:07 proc_lsst.multi:146 [INFO] found job 16912370 in provider checkpoint proc_lsst.multi INFO: found job 16912370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:07 proc_lsst.multi:146 [INFO] found job 16912371 in provider checkpoint proc_lsst.multi INFO: found job 16912371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:07 proc_lsst.multi:146 [INFO] found job 16912372 in provider checkpoint proc_lsst.multi INFO: found job 16912372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:07 proc_lsst.multi:146 [INFO] found job 16912373 in provider checkpoint proc_lsst.multi INFO: found job 16912373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:07 proc_lsst.multi:146 [INFO] found job 16912374 in provider checkpoint proc_lsst.multi INFO: found job 16912374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:07 proc_lsst.multi:146 [INFO] found job 16912375 in provider checkpoint proc_lsst.multi INFO: found job 16912375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:08 proc_lsst.multi:146 [INFO] found job 16912378 in provider checkpoint proc_lsst.multi INFO: found job 16912378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 2024-03-08 07:26:28 proc_lsst.multi:146 [INFO] found job 75775 in provider local proc_lsst.multi INFO: found job 75775 in provider local 2024-03-08 07:26:28 proc_lsst.multi:146 [INFO] found job 16912283 in provider astro proc_lsst.multi INFO: found job 16912283 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:28 proc_lsst.multi:146 [INFO] found job 16912284 in provider astro proc_lsst.multi INFO: found job 16912284 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:28 proc_lsst.multi:146 [INFO] found job 16912285 in provider astro proc_lsst.multi INFO: found job 16912285 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:28 proc_lsst.multi:146 [INFO] found job 16912286 in provider astro proc_lsst.multi INFO: found job 16912286 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:28 proc_lsst.multi:146 [INFO] found job 16912287 in provider astro proc_lsst.multi INFO: found job 16912287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:29 proc_lsst.multi:146 [INFO] found job 16912288 in provider astro proc_lsst.multi INFO: found job 16912288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:29 proc_lsst.multi:146 [INFO] found job 16912289 in provider astro proc_lsst.multi INFO: found job 16912289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:29 proc_lsst.multi:146 [INFO] found job 16912290 in provider astro proc_lsst.multi INFO: found job 16912290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:29 proc_lsst.multi:146 [INFO] found job 16912291 in provider astro proc_lsst.multi INFO: found job 16912291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:29 proc_lsst.multi:146 [INFO] found job 16912292 in provider astro proc_lsst.multi INFO: found job 16912292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:29 proc_lsst.multi:146 [INFO] found job 16912293 in provider astro proc_lsst.multi INFO: found job 16912293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:30 proc_lsst.multi:146 [INFO] found job 16912294 in provider astro proc_lsst.multi INFO: found job 16912294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:30 proc_lsst.multi:146 [INFO] found job 16912295 in provider astro proc_lsst.multi INFO: found job 16912295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:30 proc_lsst.multi:146 [INFO] found job 16912296 in provider astro proc_lsst.multi INFO: found job 16912296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:30 proc_lsst.multi:146 [INFO] found job 16912297 in provider astro proc_lsst.multi INFO: found job 16912297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.dataflow.dflow INFO: Task 970 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca29a887279439453680fdc7b649d988 with result from task 970 parsl.dataflow.dflow DEBUG: Task 970 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 971 has memoization hash be6c12cbb0161f31deb3d26a839efcdf parsl.dataflow.memoization INFO: Task 971 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa454f8b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 971 try 0 launched on executor multi with executor id 1115 parsl.dataflow.dflow INFO: Standard output for task 971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/7296ef95-6d94-44ee-927d-647ff6a9ff1e_calibrate_855042_61.stdout parsl.dataflow.dflow INFO: Standard error for task 971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/7296ef95-6d94-44ee-927d-647ff6a9ff1e_calibrate_855042_61.stderr parsl.dataflow.dflow INFO: Standard output for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/c409a54a-f82f-433f-b837-c9fcb5c96001_characterizeImage_855042_61.stdout parsl.dataflow.dflow INFO: Standard error for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855042/c409a54a-f82f-433f-b837-c9fcb5c96001_characterizeImage_855042_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:31 proc_lsst.multi:146 [INFO] found job 16912298 in provider astro proc_lsst.multi INFO: found job 16912298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:31 proc_lsst.multi:146 [INFO] found job 16912299 in provider astro proc_lsst.multi INFO: found job 16912299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:31 proc_lsst.multi:146 [INFO] found job 16912300 in provider astro proc_lsst.multi INFO: found job 16912300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:31 proc_lsst.multi:146 [INFO] found job 16912301 in provider astro proc_lsst.multi INFO: found job 16912301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:31 proc_lsst.multi:146 [INFO] found job 16912302 in provider astro proc_lsst.multi INFO: found job 16912302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:31 proc_lsst.multi:146 [INFO] found job 16912303 in provider astro proc_lsst.multi INFO: found job 16912303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:32 proc_lsst.multi:146 [INFO] found job 16912304 in provider astro proc_lsst.multi INFO: found job 16912304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:32 proc_lsst.multi:146 [INFO] found job 16912305 in provider astro proc_lsst.multi INFO: found job 16912305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:32 proc_lsst.multi:146 [INFO] found job 16912306 in provider astro proc_lsst.multi INFO: found job 16912306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:32 proc_lsst.multi:146 [INFO] found job 16912307 in provider astro proc_lsst.multi INFO: found job 16912307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:32 proc_lsst.multi:146 [INFO] found job 16912308 in provider astro proc_lsst.multi INFO: found job 16912308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:32 proc_lsst.multi:146 [INFO] found job 16912309 in provider astro proc_lsst.multi INFO: found job 16912309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:33 proc_lsst.multi:146 [INFO] found job 16912310 in provider astro proc_lsst.multi INFO: found job 16912310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:33 proc_lsst.multi:146 [INFO] found job 16912311 in provider astro proc_lsst.multi INFO: found job 16912311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:33 proc_lsst.multi:146 [INFO] found job 16912312 in provider astro proc_lsst.multi INFO: found job 16912312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:33 proc_lsst.multi:146 [INFO] found job 16912313 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912313 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:33 proc_lsst.multi:146 [INFO] found job 16912314 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912314 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:33 proc_lsst.multi:146 [INFO] found job 16912315 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912315 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:33 proc_lsst.multi:146 [INFO] found job 16912316 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912316 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:34 proc_lsst.multi:146 [INFO] found job 16912317 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:34 proc_lsst.multi:146 [INFO] found job 16912318 in provider escience_a40 proc_lsst.multi INFO: found job 16912318 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:34 proc_lsst.multi:146 [INFO] found job 16912319 in provider escience_a40 proc_lsst.multi INFO: found job 16912319 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:34 proc_lsst.multi:146 [INFO] found job 16912320 in provider escience_a40 proc_lsst.multi INFO: found job 16912320 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:34 proc_lsst.multi:146 [INFO] found job 16912321 in provider escience_a40 proc_lsst.multi INFO: found job 16912321 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:34 proc_lsst.multi:146 [INFO] found job 16912322 in provider escience_a40 proc_lsst.multi INFO: found job 16912322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:34 proc_lsst.multi:146 [INFO] found job 16912323 in provider escience_a40 proc_lsst.multi INFO: found job 16912323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:35 proc_lsst.multi:146 [INFO] found job 16912324 in provider escience_a40 proc_lsst.multi INFO: found job 16912324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:35 proc_lsst.multi:146 [INFO] found job 16912325 in provider escience_a40 proc_lsst.multi INFO: found job 16912325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:35 proc_lsst.multi:146 [INFO] found job 16912326 in provider escience_a40 proc_lsst.multi INFO: found job 16912326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:35 proc_lsst.multi:146 [INFO] found job 16912327 in provider escience_a40 proc_lsst.multi INFO: found job 16912327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:35 proc_lsst.multi:146 [INFO] found job 16912328 in provider escience_a40 proc_lsst.multi INFO: found job 16912328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:35 proc_lsst.multi:146 [INFO] found job 16912329 in provider escience_a40 proc_lsst.multi INFO: found job 16912329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:35 proc_lsst.multi:146 [INFO] found job 16912330 in provider escience_a40 proc_lsst.multi INFO: found job 16912330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:36 proc_lsst.multi:146 [INFO] found job 16912331 in provider escience_a40 proc_lsst.multi INFO: found job 16912331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:36 proc_lsst.multi:146 [INFO] found job 16912332 in provider escience_a40 proc_lsst.multi INFO: found job 16912332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:36 proc_lsst.multi:146 [INFO] found job 16912333 in provider escience_a40 proc_lsst.multi INFO: found job 16912333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:36 proc_lsst.multi:146 [INFO] found job 16912334 in provider escience_a40 proc_lsst.multi INFO: found job 16912334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:36 proc_lsst.multi:146 [INFO] found job 16912335 in provider escience_a40 proc_lsst.multi INFO: found job 16912335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:36 proc_lsst.multi:146 [INFO] found job 16912336 in provider escience_a40 proc_lsst.multi INFO: found job 16912336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:37 proc_lsst.multi:146 [INFO] found job 16912337 in provider escience_a40 proc_lsst.multi INFO: found job 16912337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:37 proc_lsst.multi:146 [INFO] found job 16912338 in provider checkpoint proc_lsst.multi INFO: found job 16912338 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:37 proc_lsst.multi:146 [INFO] found job 16912339 in provider checkpoint proc_lsst.multi INFO: found job 16912339 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:37 proc_lsst.multi:146 [INFO] found job 16912340 in provider checkpoint proc_lsst.multi INFO: found job 16912340 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:37 proc_lsst.multi:146 [INFO] found job 16912341 in provider checkpoint proc_lsst.multi INFO: found job 16912341 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:37 proc_lsst.multi:146 [INFO] found job 16912342 in provider checkpoint proc_lsst.multi INFO: found job 16912342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:37 proc_lsst.multi:146 [INFO] found job 16912343 in provider checkpoint proc_lsst.multi INFO: found job 16912343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:38 proc_lsst.multi:146 [INFO] found job 16912344 in provider checkpoint proc_lsst.multi INFO: found job 16912344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:38 proc_lsst.multi:146 [INFO] found job 16912345 in provider checkpoint proc_lsst.multi INFO: found job 16912345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:38 proc_lsst.multi:146 [INFO] found job 16912346 in provider checkpoint proc_lsst.multi INFO: found job 16912346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:38 proc_lsst.multi:146 [INFO] found job 16912347 in provider checkpoint proc_lsst.multi INFO: found job 16912347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:38 proc_lsst.multi:146 [INFO] found job 16912348 in provider checkpoint proc_lsst.multi INFO: found job 16912348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:38 proc_lsst.multi:146 [INFO] found job 16912349 in provider checkpoint proc_lsst.multi INFO: found job 16912349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:39 proc_lsst.multi:146 [INFO] found job 16912350 in provider checkpoint proc_lsst.multi INFO: found job 16912350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:39 proc_lsst.multi:146 [INFO] found job 16912351 in provider checkpoint proc_lsst.multi INFO: found job 16912351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:39 proc_lsst.multi:146 [INFO] found job 16912352 in provider checkpoint proc_lsst.multi INFO: found job 16912352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:39 proc_lsst.multi:146 [INFO] found job 16912353 in provider checkpoint proc_lsst.multi INFO: found job 16912353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:39 proc_lsst.multi:146 [INFO] found job 16912354 in provider checkpoint proc_lsst.multi INFO: found job 16912354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:39 proc_lsst.multi:146 [INFO] found job 16912355 in provider checkpoint proc_lsst.multi INFO: found job 16912355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:40 proc_lsst.multi:146 [INFO] found job 16912356 in provider checkpoint proc_lsst.multi INFO: found job 16912356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:40 proc_lsst.multi:146 [INFO] found job 16912359 in provider checkpoint proc_lsst.multi INFO: found job 16912359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:40 proc_lsst.multi:146 [INFO] found job 16912360 in provider checkpoint proc_lsst.multi INFO: found job 16912360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:40 proc_lsst.multi:146 [INFO] found job 16912361 in provider checkpoint proc_lsst.multi INFO: found job 16912361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:40 proc_lsst.multi:146 [INFO] found job 16912362 in provider checkpoint proc_lsst.multi INFO: found job 16912362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:40 proc_lsst.multi:146 [INFO] found job 16912363 in provider checkpoint proc_lsst.multi INFO: found job 16912363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:40 proc_lsst.multi:146 [INFO] found job 16912364 in provider checkpoint proc_lsst.multi INFO: found job 16912364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:41 proc_lsst.multi:146 [INFO] found job 16912365 in provider checkpoint proc_lsst.multi INFO: found job 16912365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:41 proc_lsst.multi:146 [INFO] found job 16912366 in provider checkpoint proc_lsst.multi INFO: found job 16912366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:41 proc_lsst.multi:146 [INFO] found job 16912367 in provider checkpoint proc_lsst.multi INFO: found job 16912367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:41 proc_lsst.multi:146 [INFO] found job 16912368 in provider checkpoint proc_lsst.multi INFO: found job 16912368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:41 proc_lsst.multi:146 [INFO] found job 16912369 in provider checkpoint proc_lsst.multi INFO: found job 16912369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:41 proc_lsst.multi:146 [INFO] found job 16912370 in provider checkpoint proc_lsst.multi INFO: found job 16912370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:42 proc_lsst.multi:146 [INFO] found job 16912371 in provider checkpoint proc_lsst.multi INFO: found job 16912371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:42 proc_lsst.multi:146 [INFO] found job 16912372 in provider checkpoint proc_lsst.multi INFO: found job 16912372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:42 proc_lsst.multi:146 [INFO] found job 16912373 in provider checkpoint proc_lsst.multi INFO: found job 16912373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:42 proc_lsst.multi:146 [INFO] found job 16912374 in provider checkpoint proc_lsst.multi INFO: found job 16912374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:42 proc_lsst.multi:146 [INFO] found job 16912375 in provider checkpoint proc_lsst.multi INFO: found job 16912375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:26:42 proc_lsst.multi:146 [INFO] found job 16912378 in provider checkpoint proc_lsst.multi INFO: found job 16912378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 2024-03-08 07:27:02 proc_lsst.multi:146 [INFO] found job 75775 in provider local proc_lsst.multi INFO: found job 75775 in provider local 2024-03-08 07:27:02 proc_lsst.multi:146 [INFO] found job 16912283 in provider astro proc_lsst.multi INFO: found job 16912283 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:03 proc_lsst.multi:146 [INFO] found job 16912284 in provider astro proc_lsst.multi INFO: found job 16912284 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:03 proc_lsst.multi:146 [INFO] found job 16912285 in provider astro proc_lsst.multi INFO: found job 16912285 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:03 proc_lsst.multi:146 [INFO] found job 16912286 in provider astro proc_lsst.multi INFO: found job 16912286 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:03 proc_lsst.multi:146 [INFO] found job 16912287 in provider astro proc_lsst.multi INFO: found job 16912287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:03 proc_lsst.multi:146 [INFO] found job 16912288 in provider astro proc_lsst.multi INFO: found job 16912288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:03 proc_lsst.multi:146 [INFO] found job 16912289 in provider astro proc_lsst.multi INFO: found job 16912289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:04 proc_lsst.multi:146 [INFO] found job 16912290 in provider astro proc_lsst.multi INFO: found job 16912290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:04 proc_lsst.multi:146 [INFO] found job 16912291 in provider astro proc_lsst.multi INFO: found job 16912291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:04 proc_lsst.multi:146 [INFO] found job 16912292 in provider astro proc_lsst.multi INFO: found job 16912292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:04 proc_lsst.multi:146 [INFO] found job 16912293 in provider astro proc_lsst.multi INFO: found job 16912293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:04 proc_lsst.multi:146 [INFO] found job 16912294 in provider astro proc_lsst.multi INFO: found job 16912294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:04 proc_lsst.multi:146 [INFO] found job 16912295 in provider astro proc_lsst.multi INFO: found job 16912295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:04 proc_lsst.multi:146 [INFO] found job 16912296 in provider astro proc_lsst.multi INFO: found job 16912296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:05 proc_lsst.multi:146 [INFO] found job 16912297 in provider astro proc_lsst.multi INFO: found job 16912297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:05 proc_lsst.multi:146 [INFO] found job 16912298 in provider astro proc_lsst.multi INFO: found job 16912298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:05 proc_lsst.multi:146 [INFO] found job 16912299 in provider astro proc_lsst.multi INFO: found job 16912299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:05 proc_lsst.multi:146 [INFO] found job 16912300 in provider astro proc_lsst.multi INFO: found job 16912300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:05 proc_lsst.multi:146 [INFO] found job 16912301 in provider astro proc_lsst.multi INFO: found job 16912301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:05 proc_lsst.multi:146 [INFO] found job 16912302 in provider astro proc_lsst.multi INFO: found job 16912302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:05 proc_lsst.multi:146 [INFO] found job 16912303 in provider astro proc_lsst.multi INFO: found job 16912303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:06 proc_lsst.multi:146 [INFO] found job 16912304 in provider astro proc_lsst.multi INFO: found job 16912304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:06 proc_lsst.multi:146 [INFO] found job 16912305 in provider astro proc_lsst.multi INFO: found job 16912305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:06 proc_lsst.multi:146 [INFO] found job 16912306 in provider astro proc_lsst.multi INFO: found job 16912306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:06 proc_lsst.multi:146 [INFO] found job 16912307 in provider astro proc_lsst.multi INFO: found job 16912307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:06 proc_lsst.multi:146 [INFO] found job 16912308 in provider astro proc_lsst.multi INFO: found job 16912308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:06 proc_lsst.multi:146 [INFO] found job 16912309 in provider astro proc_lsst.multi INFO: found job 16912309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:06 proc_lsst.multi:146 [INFO] found job 16912310 in provider astro proc_lsst.multi INFO: found job 16912310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:07 proc_lsst.multi:146 [INFO] found job 16912311 in provider astro proc_lsst.multi INFO: found job 16912311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:07 proc_lsst.multi:146 [INFO] found job 16912312 in provider astro proc_lsst.multi INFO: found job 16912312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:07 proc_lsst.multi:146 [INFO] found job 16912313 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912313 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:07 proc_lsst.multi:146 [INFO] found job 16912314 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912314 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:07 proc_lsst.multi:146 [INFO] found job 16912315 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912315 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:07 proc_lsst.multi:146 [INFO] found job 16912316 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912316 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:07 proc_lsst.multi:146 [INFO] found job 16912317 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:08 proc_lsst.multi:146 [INFO] found job 16912318 in provider escience_a40 proc_lsst.multi INFO: found job 16912318 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:08 proc_lsst.multi:146 [INFO] found job 16912319 in provider escience_a40 proc_lsst.multi INFO: found job 16912319 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:08 proc_lsst.multi:146 [INFO] found job 16912320 in provider escience_a40 proc_lsst.multi INFO: found job 16912320 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:08 proc_lsst.multi:146 [INFO] found job 16912321 in provider escience_a40 proc_lsst.multi INFO: found job 16912321 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:08 proc_lsst.multi:146 [INFO] found job 16912322 in provider escience_a40 proc_lsst.multi INFO: found job 16912322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:08 proc_lsst.multi:146 [INFO] found job 16912323 in provider escience_a40 proc_lsst.multi INFO: found job 16912323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:09 proc_lsst.multi:146 [INFO] found job 16912324 in provider escience_a40 proc_lsst.multi INFO: found job 16912324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:09 proc_lsst.multi:146 [INFO] found job 16912325 in provider escience_a40 proc_lsst.multi INFO: found job 16912325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:09 proc_lsst.multi:146 [INFO] found job 16912326 in provider escience_a40 proc_lsst.multi INFO: found job 16912326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:09 proc_lsst.multi:146 [INFO] found job 16912327 in provider escience_a40 proc_lsst.multi INFO: found job 16912327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:09 proc_lsst.multi:146 [INFO] found job 16912328 in provider escience_a40 proc_lsst.multi INFO: found job 16912328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:09 proc_lsst.multi:146 [INFO] found job 16912329 in provider escience_a40 proc_lsst.multi INFO: found job 16912329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:09 proc_lsst.multi:146 [INFO] found job 16912330 in provider escience_a40 proc_lsst.multi INFO: found job 16912330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:10 proc_lsst.multi:146 [INFO] found job 16912331 in provider escience_a40 proc_lsst.multi INFO: found job 16912331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:10 proc_lsst.multi:146 [INFO] found job 16912332 in provider escience_a40 proc_lsst.multi INFO: found job 16912332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:10 proc_lsst.multi:146 [INFO] found job 16912333 in provider escience_a40 proc_lsst.multi INFO: found job 16912333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:10 proc_lsst.multi:146 [INFO] found job 16912334 in provider escience_a40 proc_lsst.multi INFO: found job 16912334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:10 proc_lsst.multi:146 [INFO] found job 16912335 in provider escience_a40 proc_lsst.multi INFO: found job 16912335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:10 proc_lsst.multi:146 [INFO] found job 16912336 in provider escience_a40 proc_lsst.multi INFO: found job 16912336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:10 proc_lsst.multi:146 [INFO] found job 16912337 in provider escience_a40 proc_lsst.multi INFO: found job 16912337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:11 proc_lsst.multi:146 [INFO] found job 16912338 in provider checkpoint proc_lsst.multi INFO: found job 16912338 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:11 proc_lsst.multi:146 [INFO] found job 16912339 in provider checkpoint proc_lsst.multi INFO: found job 16912339 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:11 proc_lsst.multi:146 [INFO] found job 16912340 in provider checkpoint proc_lsst.multi INFO: found job 16912340 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:11 proc_lsst.multi:146 [INFO] found job 16912341 in provider checkpoint proc_lsst.multi INFO: found job 16912341 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:11 proc_lsst.multi:146 [INFO] found job 16912342 in provider checkpoint proc_lsst.multi INFO: found job 16912342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:11 proc_lsst.multi:146 [INFO] found job 16912343 in provider checkpoint proc_lsst.multi INFO: found job 16912343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:11 proc_lsst.multi:146 [INFO] found job 16912344 in provider checkpoint proc_lsst.multi INFO: found job 16912344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:12 proc_lsst.multi:146 [INFO] found job 16912345 in provider checkpoint proc_lsst.multi INFO: found job 16912345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:12 proc_lsst.multi:146 [INFO] found job 16912346 in provider checkpoint proc_lsst.multi INFO: found job 16912346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:12 proc_lsst.multi:146 [INFO] found job 16912347 in provider checkpoint proc_lsst.multi INFO: found job 16912347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:12 proc_lsst.multi:146 [INFO] found job 16912348 in provider checkpoint proc_lsst.multi INFO: found job 16912348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:12 proc_lsst.multi:146 [INFO] found job 16912349 in provider checkpoint proc_lsst.multi INFO: found job 16912349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:12 proc_lsst.multi:146 [INFO] found job 16912350 in provider checkpoint proc_lsst.multi INFO: found job 16912350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:12 proc_lsst.multi:146 [INFO] found job 16912351 in provider checkpoint proc_lsst.multi INFO: found job 16912351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:13 proc_lsst.multi:146 [INFO] found job 16912352 in provider checkpoint proc_lsst.multi INFO: found job 16912352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:13 proc_lsst.multi:146 [INFO] found job 16912353 in provider checkpoint proc_lsst.multi INFO: found job 16912353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:13 proc_lsst.multi:146 [INFO] found job 16912354 in provider checkpoint proc_lsst.multi INFO: found job 16912354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:13 proc_lsst.multi:146 [INFO] found job 16912355 in provider checkpoint proc_lsst.multi INFO: found job 16912355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:13 proc_lsst.multi:146 [INFO] found job 16912356 in provider checkpoint proc_lsst.multi INFO: found job 16912356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:13 proc_lsst.multi:146 [INFO] found job 16912359 in provider checkpoint proc_lsst.multi INFO: found job 16912359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:13 proc_lsst.multi:146 [INFO] found job 16912360 in provider checkpoint proc_lsst.multi INFO: found job 16912360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:14 proc_lsst.multi:146 [INFO] found job 16912361 in provider checkpoint proc_lsst.multi INFO: found job 16912361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:14 proc_lsst.multi:146 [INFO] found job 16912362 in provider checkpoint proc_lsst.multi INFO: found job 16912362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:14 proc_lsst.multi:146 [INFO] found job 16912363 in provider checkpoint proc_lsst.multi INFO: found job 16912363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:14 proc_lsst.multi:146 [INFO] found job 16912364 in provider checkpoint proc_lsst.multi INFO: found job 16912364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:14 proc_lsst.multi:146 [INFO] found job 16912365 in provider checkpoint proc_lsst.multi INFO: found job 16912365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:14 proc_lsst.multi:146 [INFO] found job 16912366 in provider checkpoint proc_lsst.multi INFO: found job 16912366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:15 proc_lsst.multi:146 [INFO] found job 16912367 in provider checkpoint proc_lsst.multi INFO: found job 16912367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:15 proc_lsst.multi:146 [INFO] found job 16912368 in provider checkpoint proc_lsst.multi INFO: found job 16912368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:15 proc_lsst.multi:146 [INFO] found job 16912369 in provider checkpoint proc_lsst.multi INFO: found job 16912369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:15 proc_lsst.multi:146 [INFO] found job 16912370 in provider checkpoint proc_lsst.multi INFO: found job 16912370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:15 proc_lsst.multi:146 [INFO] found job 16912371 in provider checkpoint proc_lsst.multi INFO: found job 16912371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:15 proc_lsst.multi:146 [INFO] found job 16912372 in provider checkpoint proc_lsst.multi INFO: found job 16912372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:15 proc_lsst.multi:146 [INFO] found job 16912373 in provider checkpoint proc_lsst.multi INFO: found job 16912373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:16 proc_lsst.multi:146 [INFO] found job 16912374 in provider checkpoint proc_lsst.multi INFO: found job 16912374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:16 proc_lsst.multi:146 [INFO] found job 16912375 in provider checkpoint proc_lsst.multi INFO: found job 16912375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:16 proc_lsst.multi:146 [INFO] found job 16912378 in provider checkpoint proc_lsst.multi INFO: found job 16912378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.dflow INFO: Task 971 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be6c12cbb0161f31deb3d26a839efcdf with result from task 971 parsl.dataflow.dflow DEBUG: Task 971 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/7296ef95-6d94-44ee-927d-647ff6a9ff1e_calibrate_855042_61.stdout parsl.dataflow.dflow INFO: Standard error for task 971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855042/7296ef95-6d94-44ee-927d-647ff6a9ff1e_calibrate_855042_61.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 2024-03-08 07:27:36 proc_lsst.multi:146 [INFO] found job 75775 in provider local proc_lsst.multi INFO: found job 75775 in provider local 2024-03-08 07:27:36 proc_lsst.multi:146 [INFO] found job 16912283 in provider astro proc_lsst.multi INFO: found job 16912283 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:36 proc_lsst.multi:146 [INFO] found job 16912284 in provider astro proc_lsst.multi INFO: found job 16912284 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:36 proc_lsst.multi:146 [INFO] found job 16912285 in provider astro proc_lsst.multi INFO: found job 16912285 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:37 proc_lsst.multi:146 [INFO] found job 16912286 in provider astro proc_lsst.multi INFO: found job 16912286 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:37 proc_lsst.multi:146 [INFO] found job 16912287 in provider astro proc_lsst.multi INFO: found job 16912287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:37 proc_lsst.multi:146 [INFO] found job 16912288 in provider astro proc_lsst.multi INFO: found job 16912288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:37 proc_lsst.multi:146 [INFO] found job 16912289 in provider astro proc_lsst.multi INFO: found job 16912289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:37 proc_lsst.multi:146 [INFO] found job 16912290 in provider astro proc_lsst.multi INFO: found job 16912290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:37 proc_lsst.multi:146 [INFO] found job 16912291 in provider astro proc_lsst.multi INFO: found job 16912291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:38 proc_lsst.multi:146 [INFO] found job 16912292 in provider astro proc_lsst.multi INFO: found job 16912292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:38 proc_lsst.multi:146 [INFO] found job 16912293 in provider astro proc_lsst.multi INFO: found job 16912293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:38 proc_lsst.multi:146 [INFO] found job 16912294 in provider astro proc_lsst.multi INFO: found job 16912294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:38 proc_lsst.multi:146 [INFO] found job 16912295 in provider astro proc_lsst.multi INFO: found job 16912295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:38 proc_lsst.multi:146 [INFO] found job 16912296 in provider astro proc_lsst.multi INFO: found job 16912296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:38 proc_lsst.multi:146 [INFO] found job 16912297 in provider astro proc_lsst.multi INFO: found job 16912297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:38 proc_lsst.multi:146 [INFO] found job 16912298 in provider astro proc_lsst.multi INFO: found job 16912298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:39 proc_lsst.multi:146 [INFO] found job 16912299 in provider astro proc_lsst.multi INFO: found job 16912299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:39 proc_lsst.multi:146 [INFO] found job 16912300 in provider astro proc_lsst.multi INFO: found job 16912300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:39 proc_lsst.multi:146 [INFO] found job 16912301 in provider astro proc_lsst.multi INFO: found job 16912301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:39 proc_lsst.multi:146 [INFO] found job 16912302 in provider astro proc_lsst.multi INFO: found job 16912302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:39 proc_lsst.multi:146 [INFO] found job 16912303 in provider astro proc_lsst.multi INFO: found job 16912303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:39 proc_lsst.multi:146 [INFO] found job 16912304 in provider astro proc_lsst.multi INFO: found job 16912304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:39 proc_lsst.multi:146 [INFO] found job 16912305 in provider astro proc_lsst.multi INFO: found job 16912305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:40 proc_lsst.multi:146 [INFO] found job 16912306 in provider astro proc_lsst.multi INFO: found job 16912306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:40 proc_lsst.multi:146 [INFO] found job 16912307 in provider astro proc_lsst.multi INFO: found job 16912307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:40 proc_lsst.multi:146 [INFO] found job 16912308 in provider astro proc_lsst.multi INFO: found job 16912308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:40 proc_lsst.multi:146 [INFO] found job 16912309 in provider astro proc_lsst.multi INFO: found job 16912309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:40 proc_lsst.multi:146 [INFO] found job 16912310 in provider astro proc_lsst.multi INFO: found job 16912310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:40 proc_lsst.multi:146 [INFO] found job 16912311 in provider astro proc_lsst.multi INFO: found job 16912311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:41 proc_lsst.multi:146 [INFO] found job 16912312 in provider astro proc_lsst.multi INFO: found job 16912312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:41 proc_lsst.multi:146 [INFO] found job 16912313 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912313 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:41 proc_lsst.multi:146 [INFO] found job 16912314 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912314 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:41 proc_lsst.multi:146 [INFO] found job 16912315 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912315 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:41 proc_lsst.multi:146 [INFO] found job 16912316 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912316 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:41 proc_lsst.multi:146 [INFO] found job 16912317 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:42 proc_lsst.multi:146 [INFO] found job 16912318 in provider escience_a40 proc_lsst.multi INFO: found job 16912318 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:42 proc_lsst.multi:146 [INFO] found job 16912319 in provider escience_a40 proc_lsst.multi INFO: found job 16912319 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:42 proc_lsst.multi:146 [INFO] found job 16912320 in provider escience_a40 proc_lsst.multi INFO: found job 16912320 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:42 proc_lsst.multi:146 [INFO] found job 16912321 in provider escience_a40 proc_lsst.multi INFO: found job 16912321 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:42 proc_lsst.multi:146 [INFO] found job 16912322 in provider escience_a40 proc_lsst.multi INFO: found job 16912322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:42 proc_lsst.multi:146 [INFO] found job 16912323 in provider escience_a40 proc_lsst.multi INFO: found job 16912323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:42 proc_lsst.multi:146 [INFO] found job 16912324 in provider escience_a40 proc_lsst.multi INFO: found job 16912324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:43 proc_lsst.multi:146 [INFO] found job 16912325 in provider escience_a40 proc_lsst.multi INFO: found job 16912325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:43 proc_lsst.multi:146 [INFO] found job 16912326 in provider escience_a40 proc_lsst.multi INFO: found job 16912326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:43 proc_lsst.multi:146 [INFO] found job 16912327 in provider escience_a40 proc_lsst.multi INFO: found job 16912327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:43 proc_lsst.multi:146 [INFO] found job 16912328 in provider escience_a40 proc_lsst.multi INFO: found job 16912328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:43 proc_lsst.multi:146 [INFO] found job 16912329 in provider escience_a40 proc_lsst.multi INFO: found job 16912329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:43 proc_lsst.multi:146 [INFO] found job 16912330 in provider escience_a40 proc_lsst.multi INFO: found job 16912330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:43 proc_lsst.multi:146 [INFO] found job 16912331 in provider escience_a40 proc_lsst.multi INFO: found job 16912331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:44 proc_lsst.multi:146 [INFO] found job 16912332 in provider escience_a40 proc_lsst.multi INFO: found job 16912332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:44 proc_lsst.multi:146 [INFO] found job 16912333 in provider escience_a40 proc_lsst.multi INFO: found job 16912333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:44 proc_lsst.multi:146 [INFO] found job 16912334 in provider escience_a40 proc_lsst.multi INFO: found job 16912334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:44 proc_lsst.multi:146 [INFO] found job 16912335 in provider escience_a40 proc_lsst.multi INFO: found job 16912335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:44 proc_lsst.multi:146 [INFO] found job 16912336 in provider escience_a40 proc_lsst.multi INFO: found job 16912336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:44 proc_lsst.multi:146 [INFO] found job 16912337 in provider escience_a40 proc_lsst.multi INFO: found job 16912337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:44 proc_lsst.multi:146 [INFO] found job 16912338 in provider checkpoint proc_lsst.multi INFO: found job 16912338 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:45 proc_lsst.multi:146 [INFO] found job 16912339 in provider checkpoint proc_lsst.multi INFO: found job 16912339 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:45 proc_lsst.multi:146 [INFO] found job 16912340 in provider checkpoint proc_lsst.multi INFO: found job 16912340 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:45 proc_lsst.multi:146 [INFO] found job 16912341 in provider checkpoint proc_lsst.multi INFO: found job 16912341 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:45 proc_lsst.multi:146 [INFO] found job 16912342 in provider checkpoint proc_lsst.multi INFO: found job 16912342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:45 proc_lsst.multi:146 [INFO] found job 16912343 in provider checkpoint proc_lsst.multi INFO: found job 16912343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:45 proc_lsst.multi:146 [INFO] found job 16912344 in provider checkpoint proc_lsst.multi INFO: found job 16912344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:46 proc_lsst.multi:146 [INFO] found job 16912345 in provider checkpoint proc_lsst.multi INFO: found job 16912345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:46 proc_lsst.multi:146 [INFO] found job 16912346 in provider checkpoint proc_lsst.multi INFO: found job 16912346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:46 proc_lsst.multi:146 [INFO] found job 16912347 in provider checkpoint proc_lsst.multi INFO: found job 16912347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:46 proc_lsst.multi:146 [INFO] found job 16912348 in provider checkpoint proc_lsst.multi INFO: found job 16912348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:46 proc_lsst.multi:146 [INFO] found job 16912349 in provider checkpoint proc_lsst.multi INFO: found job 16912349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:46 proc_lsst.multi:146 [INFO] found job 16912350 in provider checkpoint proc_lsst.multi INFO: found job 16912350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:46 proc_lsst.multi:146 [INFO] found job 16912351 in provider checkpoint proc_lsst.multi INFO: found job 16912351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:47 proc_lsst.multi:146 [INFO] found job 16912352 in provider checkpoint proc_lsst.multi INFO: found job 16912352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:47 proc_lsst.multi:146 [INFO] found job 16912353 in provider checkpoint proc_lsst.multi INFO: found job 16912353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:47 proc_lsst.multi:146 [INFO] found job 16912354 in provider checkpoint proc_lsst.multi INFO: found job 16912354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:47 proc_lsst.multi:146 [INFO] found job 16912355 in provider checkpoint proc_lsst.multi INFO: found job 16912355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:47 proc_lsst.multi:146 [INFO] found job 16912356 in provider checkpoint proc_lsst.multi INFO: found job 16912356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:47 proc_lsst.multi:146 [INFO] found job 16912359 in provider checkpoint proc_lsst.multi INFO: found job 16912359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:47 proc_lsst.multi:146 [INFO] found job 16912360 in provider checkpoint proc_lsst.multi INFO: found job 16912360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:48 proc_lsst.multi:146 [INFO] found job 16912361 in provider checkpoint proc_lsst.multi INFO: found job 16912361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:48 proc_lsst.multi:146 [INFO] found job 16912362 in provider checkpoint proc_lsst.multi INFO: found job 16912362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:48 proc_lsst.multi:146 [INFO] found job 16912363 in provider checkpoint proc_lsst.multi INFO: found job 16912363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:48 proc_lsst.multi:146 [INFO] found job 16912364 in provider checkpoint proc_lsst.multi INFO: found job 16912364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:48 proc_lsst.multi:146 [INFO] found job 16912365 in provider checkpoint proc_lsst.multi INFO: found job 16912365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:48 proc_lsst.multi:146 [INFO] found job 16912366 in provider checkpoint proc_lsst.multi INFO: found job 16912366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:48 proc_lsst.multi:146 [INFO] found job 16912367 in provider checkpoint proc_lsst.multi INFO: found job 16912367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:49 proc_lsst.multi:146 [INFO] found job 16912368 in provider checkpoint proc_lsst.multi INFO: found job 16912368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:49 proc_lsst.multi:146 [INFO] found job 16912369 in provider checkpoint proc_lsst.multi INFO: found job 16912369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:49 proc_lsst.multi:146 [INFO] found job 16912370 in provider checkpoint proc_lsst.multi INFO: found job 16912370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:49 proc_lsst.multi:146 [INFO] found job 16912371 in provider checkpoint proc_lsst.multi INFO: found job 16912371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:49 proc_lsst.multi:146 [INFO] found job 16912372 in provider checkpoint proc_lsst.multi INFO: found job 16912372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:49 proc_lsst.multi:146 [INFO] found job 16912373 in provider checkpoint proc_lsst.multi INFO: found job 16912373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:49 proc_lsst.multi:146 [INFO] found job 16912374 in provider checkpoint proc_lsst.multi INFO: found job 16912374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:50 proc_lsst.multi:146 [INFO] found job 16912375 in provider checkpoint proc_lsst.multi INFO: found job 16912375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:27:50 proc_lsst.multi:146 [INFO] found job 16912378 in provider checkpoint proc_lsst.multi INFO: found job 16912378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 2024-03-08 07:28:10 proc_lsst.multi:146 [INFO] found job 75775 in provider local proc_lsst.multi INFO: found job 75775 in provider local 2024-03-08 07:28:10 proc_lsst.multi:146 [INFO] found job 16912283 in provider astro proc_lsst.multi INFO: found job 16912283 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:10 proc_lsst.multi:146 [INFO] found job 16912284 in provider astro proc_lsst.multi INFO: found job 16912284 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:10 proc_lsst.multi:146 [INFO] found job 16912285 in provider astro proc_lsst.multi INFO: found job 16912285 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:10 proc_lsst.multi:146 [INFO] found job 16912286 in provider astro proc_lsst.multi INFO: found job 16912286 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:11 proc_lsst.multi:146 [INFO] found job 16912287 in provider astro proc_lsst.multi INFO: found job 16912287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:11 proc_lsst.multi:146 [INFO] found job 16912288 in provider astro proc_lsst.multi INFO: found job 16912288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:11 proc_lsst.multi:146 [INFO] found job 16912289 in provider astro proc_lsst.multi INFO: found job 16912289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:11 proc_lsst.multi:146 [INFO] found job 16912290 in provider astro proc_lsst.multi INFO: found job 16912290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:11 proc_lsst.multi:146 [INFO] found job 16912291 in provider astro proc_lsst.multi INFO: found job 16912291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:11 proc_lsst.multi:146 [INFO] found job 16912292 in provider astro proc_lsst.multi INFO: found job 16912292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:11 proc_lsst.multi:146 [INFO] found job 16912293 in provider astro proc_lsst.multi INFO: found job 16912293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:12 proc_lsst.multi:146 [INFO] found job 16912294 in provider astro proc_lsst.multi INFO: found job 16912294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:12 proc_lsst.multi:146 [INFO] found job 16912295 in provider astro proc_lsst.multi INFO: found job 16912295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:12 proc_lsst.multi:146 [INFO] found job 16912296 in provider astro proc_lsst.multi INFO: found job 16912296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:12 proc_lsst.multi:146 [INFO] found job 16912297 in provider astro proc_lsst.multi INFO: found job 16912297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:12 proc_lsst.multi:146 [INFO] found job 16912298 in provider astro proc_lsst.multi INFO: found job 16912298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:13 proc_lsst.multi:146 [INFO] found job 16912299 in provider astro proc_lsst.multi INFO: found job 16912299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:13 proc_lsst.multi:146 [INFO] found job 16912300 in provider astro proc_lsst.multi INFO: found job 16912300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:13 proc_lsst.multi:146 [INFO] found job 16912301 in provider astro proc_lsst.multi INFO: found job 16912301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:13 proc_lsst.multi:146 [INFO] found job 16912302 in provider astro proc_lsst.multi INFO: found job 16912302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:13 proc_lsst.multi:146 [INFO] found job 16912303 in provider astro proc_lsst.multi INFO: found job 16912303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:13 proc_lsst.multi:146 [INFO] found job 16912304 in provider astro proc_lsst.multi INFO: found job 16912304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:14 proc_lsst.multi:146 [INFO] found job 16912305 in provider astro proc_lsst.multi INFO: found job 16912305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:14 proc_lsst.multi:146 [INFO] found job 16912306 in provider astro proc_lsst.multi INFO: found job 16912306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:14 proc_lsst.multi:146 [INFO] found job 16912307 in provider astro proc_lsst.multi INFO: found job 16912307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:14 proc_lsst.multi:146 [INFO] found job 16912308 in provider astro proc_lsst.multi INFO: found job 16912308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:14 proc_lsst.multi:146 [INFO] found job 16912309 in provider astro proc_lsst.multi INFO: found job 16912309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:14 proc_lsst.multi:146 [INFO] found job 16912310 in provider astro proc_lsst.multi INFO: found job 16912310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:14 proc_lsst.multi:146 [INFO] found job 16912311 in provider astro proc_lsst.multi INFO: found job 16912311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:15 proc_lsst.multi:146 [INFO] found job 16912312 in provider astro proc_lsst.multi INFO: found job 16912312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:15 proc_lsst.multi:146 [INFO] found job 16912313 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912313 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:15 proc_lsst.multi:146 [INFO] found job 16912314 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912314 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:15 proc_lsst.multi:146 [INFO] found job 16912315 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912315 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:15 proc_lsst.multi:146 [INFO] found job 16912316 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912316 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:15 proc_lsst.multi:146 [INFO] found job 16912317 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:15 proc_lsst.multi:146 [INFO] found job 16912318 in provider escience_a40 proc_lsst.multi INFO: found job 16912318 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:16 proc_lsst.multi:146 [INFO] found job 16912319 in provider escience_a40 proc_lsst.multi INFO: found job 16912319 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:16 proc_lsst.multi:146 [INFO] found job 16912320 in provider escience_a40 proc_lsst.multi INFO: found job 16912320 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:16 proc_lsst.multi:146 [INFO] found job 16912321 in provider escience_a40 proc_lsst.multi INFO: found job 16912321 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:16 proc_lsst.multi:146 [INFO] found job 16912322 in provider escience_a40 proc_lsst.multi INFO: found job 16912322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:16 proc_lsst.multi:146 [INFO] found job 16912323 in provider escience_a40 proc_lsst.multi INFO: found job 16912323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:16 proc_lsst.multi:146 [INFO] found job 16912324 in provider escience_a40 proc_lsst.multi INFO: found job 16912324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:16 proc_lsst.multi:146 [INFO] found job 16912325 in provider escience_a40 proc_lsst.multi INFO: found job 16912325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:17 proc_lsst.multi:146 [INFO] found job 16912326 in provider escience_a40 proc_lsst.multi INFO: found job 16912326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:17 proc_lsst.multi:146 [INFO] found job 16912327 in provider escience_a40 proc_lsst.multi INFO: found job 16912327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:17 proc_lsst.multi:146 [INFO] found job 16912328 in provider escience_a40 proc_lsst.multi INFO: found job 16912328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:17 proc_lsst.multi:146 [INFO] found job 16912329 in provider escience_a40 proc_lsst.multi INFO: found job 16912329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:17 proc_lsst.multi:146 [INFO] found job 16912330 in provider escience_a40 proc_lsst.multi INFO: found job 16912330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:17 proc_lsst.multi:146 [INFO] found job 16912331 in provider escience_a40 proc_lsst.multi INFO: found job 16912331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:18 proc_lsst.multi:146 [INFO] found job 16912332 in provider escience_a40 proc_lsst.multi INFO: found job 16912332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:18 proc_lsst.multi:146 [INFO] found job 16912333 in provider escience_a40 proc_lsst.multi INFO: found job 16912333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:18 proc_lsst.multi:146 [INFO] found job 16912334 in provider escience_a40 proc_lsst.multi INFO: found job 16912334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:18 proc_lsst.multi:146 [INFO] found job 16912335 in provider escience_a40 proc_lsst.multi INFO: found job 16912335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:18 proc_lsst.multi:146 [INFO] found job 16912336 in provider escience_a40 proc_lsst.multi INFO: found job 16912336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:18 proc_lsst.multi:146 [INFO] found job 16912337 in provider escience_a40 proc_lsst.multi INFO: found job 16912337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:18 proc_lsst.multi:146 [INFO] found job 16912338 in provider checkpoint proc_lsst.multi INFO: found job 16912338 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:19 proc_lsst.multi:146 [INFO] found job 16912339 in provider checkpoint proc_lsst.multi INFO: found job 16912339 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:19 proc_lsst.multi:146 [INFO] found job 16912340 in provider checkpoint proc_lsst.multi INFO: found job 16912340 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:19 proc_lsst.multi:146 [INFO] found job 16912341 in provider checkpoint proc_lsst.multi INFO: found job 16912341 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:19 proc_lsst.multi:146 [INFO] found job 16912342 in provider checkpoint proc_lsst.multi INFO: found job 16912342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:19 proc_lsst.multi:146 [INFO] found job 16912343 in provider checkpoint proc_lsst.multi INFO: found job 16912343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:19 proc_lsst.multi:146 [INFO] found job 16912344 in provider checkpoint proc_lsst.multi INFO: found job 16912344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:20 proc_lsst.multi:146 [INFO] found job 16912345 in provider checkpoint proc_lsst.multi INFO: found job 16912345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:20 proc_lsst.multi:146 [INFO] found job 16912346 in provider checkpoint proc_lsst.multi INFO: found job 16912346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:20 proc_lsst.multi:146 [INFO] found job 16912347 in provider checkpoint proc_lsst.multi INFO: found job 16912347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:20 proc_lsst.multi:146 [INFO] found job 16912348 in provider checkpoint proc_lsst.multi INFO: found job 16912348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:20 proc_lsst.multi:146 [INFO] found job 16912349 in provider checkpoint proc_lsst.multi INFO: found job 16912349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:20 proc_lsst.multi:146 [INFO] found job 16912350 in provider checkpoint proc_lsst.multi INFO: found job 16912350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:21 proc_lsst.multi:146 [INFO] found job 16912351 in provider checkpoint proc_lsst.multi INFO: found job 16912351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:21 proc_lsst.multi:146 [INFO] found job 16912352 in provider checkpoint proc_lsst.multi INFO: found job 16912352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:21 proc_lsst.multi:146 [INFO] found job 16912353 in provider checkpoint proc_lsst.multi INFO: found job 16912353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:21 proc_lsst.multi:146 [INFO] found job 16912354 in provider checkpoint proc_lsst.multi INFO: found job 16912354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:21 proc_lsst.multi:146 [INFO] found job 16912355 in provider checkpoint proc_lsst.multi INFO: found job 16912355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:21 proc_lsst.multi:146 [INFO] found job 16912356 in provider checkpoint proc_lsst.multi INFO: found job 16912356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:21 proc_lsst.multi:146 [INFO] found job 16912359 in provider checkpoint proc_lsst.multi INFO: found job 16912359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:22 proc_lsst.multi:146 [INFO] found job 16912360 in provider checkpoint proc_lsst.multi INFO: found job 16912360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:22 proc_lsst.multi:146 [INFO] found job 16912361 in provider checkpoint proc_lsst.multi INFO: found job 16912361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Task 865 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b3fa3b170e8b6e1ab4574fd6eb14e54b with result from task 865 parsl.dataflow.dflow DEBUG: Task 865 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 866 has memoization hash d0828200d4e611cd4767be34a9aa64bb parsl.dataflow.memoization INFO: Task 866 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14aa45666700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:22 proc_lsst.multi:146 [INFO] found job 16912362 in provider checkpoint proc_lsst.multi INFO: found job 16912362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.dataflow.dflow INFO: Parsl task 866 try 0 launched on executor multi with executor id 1116 parsl.dataflow.dflow INFO: Standard output for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/638505f0-3245-4c61-9d1e-6e363b15b263_calibrate_855136_61.stdout parsl.dataflow.dflow INFO: Standard error for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/638505f0-3245-4c61-9d1e-6e363b15b263_calibrate_855136_61.stderr parsl.dataflow.dflow INFO: Standard output for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/11641d94-4cd5-4f70-96d4-b32e73f70a06_characterizeImage_855136_61.stdout parsl.dataflow.dflow INFO: Standard error for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/characterizeImage/855136/11641d94-4cd5-4f70-96d4-b32e73f70a06_characterizeImage_855136_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:22 proc_lsst.multi:146 [INFO] found job 16912363 in provider checkpoint proc_lsst.multi INFO: found job 16912363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:22 proc_lsst.multi:146 [INFO] found job 16912364 in provider checkpoint proc_lsst.multi INFO: found job 16912364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:22 proc_lsst.multi:146 [INFO] found job 16912365 in provider checkpoint proc_lsst.multi INFO: found job 16912365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:22 proc_lsst.multi:146 [INFO] found job 16912366 in provider checkpoint proc_lsst.multi INFO: found job 16912366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:23 proc_lsst.multi:146 [INFO] found job 16912367 in provider checkpoint proc_lsst.multi INFO: found job 16912367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:23 proc_lsst.multi:146 [INFO] found job 16912368 in provider checkpoint proc_lsst.multi INFO: found job 16912368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:23 proc_lsst.multi:146 [INFO] found job 16912369 in provider checkpoint proc_lsst.multi INFO: found job 16912369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:23 proc_lsst.multi:146 [INFO] found job 16912370 in provider checkpoint proc_lsst.multi INFO: found job 16912370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:23 proc_lsst.multi:146 [INFO] found job 16912371 in provider checkpoint proc_lsst.multi INFO: found job 16912371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:23 proc_lsst.multi:146 [INFO] found job 16912372 in provider checkpoint proc_lsst.multi INFO: found job 16912372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:24 proc_lsst.multi:146 [INFO] found job 16912373 in provider checkpoint proc_lsst.multi INFO: found job 16912373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:24 proc_lsst.multi:146 [INFO] found job 16912374 in provider checkpoint proc_lsst.multi INFO: found job 16912374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:24 proc_lsst.multi:146 [INFO] found job 16912375 in provider checkpoint proc_lsst.multi INFO: found job 16912375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:24 proc_lsst.multi:146 [INFO] found job 16912378 in provider checkpoint proc_lsst.multi INFO: found job 16912378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 2024-03-08 07:28:44 proc_lsst.multi:146 [INFO] found job 75775 in provider local proc_lsst.multi INFO: found job 75775 in provider local 2024-03-08 07:28:44 proc_lsst.multi:146 [INFO] found job 16912283 in provider astro proc_lsst.multi INFO: found job 16912283 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:44 proc_lsst.multi:146 [INFO] found job 16912284 in provider astro proc_lsst.multi INFO: found job 16912284 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:45 proc_lsst.multi:146 [INFO] found job 16912285 in provider astro proc_lsst.multi INFO: found job 16912285 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:46 proc_lsst.multi:146 [INFO] found job 16912286 in provider astro proc_lsst.multi INFO: found job 16912286 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:46 proc_lsst.multi:146 [INFO] found job 16912287 in provider astro proc_lsst.multi INFO: found job 16912287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:46 proc_lsst.multi:146 [INFO] found job 16912288 in provider astro proc_lsst.multi INFO: found job 16912288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:46 proc_lsst.multi:146 [INFO] found job 16912289 in provider astro proc_lsst.multi INFO: found job 16912289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:46 proc_lsst.multi:146 [INFO] found job 16912290 in provider astro proc_lsst.multi INFO: found job 16912290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:47 proc_lsst.multi:146 [INFO] found job 16912291 in provider astro proc_lsst.multi INFO: found job 16912291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:47 proc_lsst.multi:146 [INFO] found job 16912292 in provider astro proc_lsst.multi INFO: found job 16912292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:47 proc_lsst.multi:146 [INFO] found job 16912293 in provider astro proc_lsst.multi INFO: found job 16912293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:47 proc_lsst.multi:146 [INFO] found job 16912294 in provider astro proc_lsst.multi INFO: found job 16912294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:47 proc_lsst.multi:146 [INFO] found job 16912295 in provider astro proc_lsst.multi INFO: found job 16912295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:47 proc_lsst.multi:146 [INFO] found job 16912296 in provider astro proc_lsst.multi INFO: found job 16912296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:48 proc_lsst.multi:146 [INFO] found job 16912297 in provider astro proc_lsst.multi INFO: found job 16912297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:48 proc_lsst.multi:146 [INFO] found job 16912298 in provider astro proc_lsst.multi INFO: found job 16912298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:48 proc_lsst.multi:146 [INFO] found job 16912299 in provider astro proc_lsst.multi INFO: found job 16912299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:48 proc_lsst.multi:146 [INFO] found job 16912300 in provider astro proc_lsst.multi INFO: found job 16912300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:48 proc_lsst.multi:146 [INFO] found job 16912301 in provider astro proc_lsst.multi INFO: found job 16912301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:48 proc_lsst.multi:146 [INFO] found job 16912302 in provider astro proc_lsst.multi INFO: found job 16912302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:48 proc_lsst.multi:146 [INFO] found job 16912303 in provider astro proc_lsst.multi INFO: found job 16912303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:49 proc_lsst.multi:146 [INFO] found job 16912304 in provider astro proc_lsst.multi INFO: found job 16912304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:49 proc_lsst.multi:146 [INFO] found job 16912305 in provider astro proc_lsst.multi INFO: found job 16912305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:49 proc_lsst.multi:146 [INFO] found job 16912306 in provider astro proc_lsst.multi INFO: found job 16912306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:49 proc_lsst.multi:146 [INFO] found job 16912307 in provider astro proc_lsst.multi INFO: found job 16912307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:49 proc_lsst.multi:146 [INFO] found job 16912308 in provider astro proc_lsst.multi INFO: found job 16912308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:49 proc_lsst.multi:146 [INFO] found job 16912309 in provider astro proc_lsst.multi INFO: found job 16912309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:49 proc_lsst.multi:146 [INFO] found job 16912310 in provider astro proc_lsst.multi INFO: found job 16912310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:50 proc_lsst.multi:146 [INFO] found job 16912311 in provider astro proc_lsst.multi INFO: found job 16912311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:50 proc_lsst.multi:146 [INFO] found job 16912312 in provider astro proc_lsst.multi INFO: found job 16912312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:50 proc_lsst.multi:146 [INFO] found job 16912313 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912313 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:50 proc_lsst.multi:146 [INFO] found job 16912314 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912314 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:50 proc_lsst.multi:146 [INFO] found job 16912315 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912315 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:50 proc_lsst.multi:146 [INFO] found job 16912316 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912316 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:50 proc_lsst.multi:146 [INFO] found job 16912317 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:51 proc_lsst.multi:146 [INFO] found job 16912318 in provider escience_a40 proc_lsst.multi INFO: found job 16912318 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:51 proc_lsst.multi:146 [INFO] found job 16912319 in provider escience_a40 proc_lsst.multi INFO: found job 16912319 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:51 proc_lsst.multi:146 [INFO] found job 16912320 in provider escience_a40 proc_lsst.multi INFO: found job 16912320 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:51 proc_lsst.multi:146 [INFO] found job 16912321 in provider escience_a40 proc_lsst.multi INFO: found job 16912321 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:51 proc_lsst.multi:146 [INFO] found job 16912322 in provider escience_a40 proc_lsst.multi INFO: found job 16912322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:51 proc_lsst.multi:146 [INFO] found job 16912323 in provider escience_a40 proc_lsst.multi INFO: found job 16912323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:51 proc_lsst.multi:146 [INFO] found job 16912324 in provider escience_a40 proc_lsst.multi INFO: found job 16912324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:52 proc_lsst.multi:146 [INFO] found job 16912325 in provider escience_a40 proc_lsst.multi INFO: found job 16912325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:52 proc_lsst.multi:146 [INFO] found job 16912326 in provider escience_a40 proc_lsst.multi INFO: found job 16912326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:52 proc_lsst.multi:146 [INFO] found job 16912327 in provider escience_a40 proc_lsst.multi INFO: found job 16912327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:52 proc_lsst.multi:146 [INFO] found job 16912328 in provider escience_a40 proc_lsst.multi INFO: found job 16912328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:52 proc_lsst.multi:146 [INFO] found job 16912329 in provider escience_a40 proc_lsst.multi INFO: found job 16912329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:52 proc_lsst.multi:146 [INFO] found job 16912330 in provider escience_a40 proc_lsst.multi INFO: found job 16912330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:52 proc_lsst.multi:146 [INFO] found job 16912331 in provider escience_a40 proc_lsst.multi INFO: found job 16912331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:53 proc_lsst.multi:146 [INFO] found job 16912332 in provider escience_a40 proc_lsst.multi INFO: found job 16912332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:53 proc_lsst.multi:146 [INFO] found job 16912333 in provider escience_a40 proc_lsst.multi INFO: found job 16912333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:53 proc_lsst.multi:146 [INFO] found job 16912334 in provider escience_a40 proc_lsst.multi INFO: found job 16912334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:53 proc_lsst.multi:146 [INFO] found job 16912335 in provider escience_a40 proc_lsst.multi INFO: found job 16912335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:53 proc_lsst.multi:146 [INFO] found job 16912336 in provider escience_a40 proc_lsst.multi INFO: found job 16912336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:53 proc_lsst.multi:146 [INFO] found job 16912337 in provider escience_a40 proc_lsst.multi INFO: found job 16912337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:53 proc_lsst.multi:146 [INFO] found job 16912338 in provider checkpoint proc_lsst.multi INFO: found job 16912338 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:54 proc_lsst.multi:146 [INFO] found job 16912339 in provider checkpoint proc_lsst.multi INFO: found job 16912339 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:54 proc_lsst.multi:146 [INFO] found job 16912340 in provider checkpoint proc_lsst.multi INFO: found job 16912340 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:54 proc_lsst.multi:146 [INFO] found job 16912341 in provider checkpoint proc_lsst.multi INFO: found job 16912341 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:54 proc_lsst.multi:146 [INFO] found job 16912342 in provider checkpoint proc_lsst.multi INFO: found job 16912342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:54 proc_lsst.multi:146 [INFO] found job 16912343 in provider checkpoint proc_lsst.multi INFO: found job 16912343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:54 proc_lsst.multi:146 [INFO] found job 16912344 in provider checkpoint proc_lsst.multi INFO: found job 16912344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:55 proc_lsst.multi:146 [INFO] found job 16912345 in provider checkpoint proc_lsst.multi INFO: found job 16912345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:55 proc_lsst.multi:146 [INFO] found job 16912346 in provider checkpoint proc_lsst.multi INFO: found job 16912346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:55 proc_lsst.multi:146 [INFO] found job 16912347 in provider checkpoint proc_lsst.multi INFO: found job 16912347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:55 proc_lsst.multi:146 [INFO] found job 16912348 in provider checkpoint proc_lsst.multi INFO: found job 16912348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:55 proc_lsst.multi:146 [INFO] found job 16912349 in provider checkpoint proc_lsst.multi INFO: found job 16912349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:55 proc_lsst.multi:146 [INFO] found job 16912350 in provider checkpoint proc_lsst.multi INFO: found job 16912350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:55 proc_lsst.multi:146 [INFO] found job 16912351 in provider checkpoint proc_lsst.multi INFO: found job 16912351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:56 proc_lsst.multi:146 [INFO] found job 16912352 in provider checkpoint proc_lsst.multi INFO: found job 16912352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:56 proc_lsst.multi:146 [INFO] found job 16912353 in provider checkpoint proc_lsst.multi INFO: found job 16912353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:56 proc_lsst.multi:146 [INFO] found job 16912354 in provider checkpoint proc_lsst.multi INFO: found job 16912354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:56 proc_lsst.multi:146 [INFO] found job 16912355 in provider checkpoint proc_lsst.multi INFO: found job 16912355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:56 proc_lsst.multi:146 [INFO] found job 16912356 in provider checkpoint proc_lsst.multi INFO: found job 16912356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:56 proc_lsst.multi:146 [INFO] found job 16912359 in provider checkpoint proc_lsst.multi INFO: found job 16912359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:56 proc_lsst.multi:146 [INFO] found job 16912360 in provider checkpoint proc_lsst.multi INFO: found job 16912360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:57 proc_lsst.multi:146 [INFO] found job 16912361 in provider checkpoint proc_lsst.multi INFO: found job 16912361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:57 proc_lsst.multi:146 [INFO] found job 16912362 in provider checkpoint proc_lsst.multi INFO: found job 16912362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:57 proc_lsst.multi:146 [INFO] found job 16912363 in provider checkpoint proc_lsst.multi INFO: found job 16912363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:57 proc_lsst.multi:146 [INFO] found job 16912364 in provider checkpoint proc_lsst.multi INFO: found job 16912364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:57 proc_lsst.multi:146 [INFO] found job 16912365 in provider checkpoint proc_lsst.multi INFO: found job 16912365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:57 proc_lsst.multi:146 [INFO] found job 16912366 in provider checkpoint proc_lsst.multi INFO: found job 16912366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:57 proc_lsst.multi:146 [INFO] found job 16912367 in provider checkpoint proc_lsst.multi INFO: found job 16912367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:58 proc_lsst.multi:146 [INFO] found job 16912368 in provider checkpoint proc_lsst.multi INFO: found job 16912368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:58 proc_lsst.multi:146 [INFO] found job 16912369 in provider checkpoint proc_lsst.multi INFO: found job 16912369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:58 proc_lsst.multi:146 [INFO] found job 16912370 in provider checkpoint proc_lsst.multi INFO: found job 16912370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:58 proc_lsst.multi:146 [INFO] found job 16912371 in provider checkpoint proc_lsst.multi INFO: found job 16912371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:58 proc_lsst.multi:146 [INFO] found job 16912372 in provider checkpoint proc_lsst.multi INFO: found job 16912372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:58 proc_lsst.multi:146 [INFO] found job 16912373 in provider checkpoint proc_lsst.multi INFO: found job 16912373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:59 proc_lsst.multi:146 [INFO] found job 16912374 in provider checkpoint proc_lsst.multi INFO: found job 16912374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:59 proc_lsst.multi:146 [INFO] found job 16912375 in provider checkpoint proc_lsst.multi INFO: found job 16912375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:28:59 proc_lsst.multi:146 [INFO] found job 16912378 in provider checkpoint proc_lsst.multi INFO: found job 16912378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 2024-03-08 07:29:19 proc_lsst.multi:146 [INFO] found job 75775 in provider local proc_lsst.multi INFO: found job 75775 in provider local 2024-03-08 07:29:19 proc_lsst.multi:146 [INFO] found job 16912283 in provider astro proc_lsst.multi INFO: found job 16912283 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:21 proc_lsst.multi:146 [INFO] found job 16912284 in provider astro proc_lsst.multi INFO: found job 16912284 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:21 proc_lsst.multi:146 [INFO] found job 16912285 in provider astro proc_lsst.multi INFO: found job 16912285 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:21 proc_lsst.multi:146 [INFO] found job 16912286 in provider astro proc_lsst.multi INFO: found job 16912286 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:22 proc_lsst.multi:146 [INFO] found job 16912287 in provider astro proc_lsst.multi INFO: found job 16912287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:22 proc_lsst.multi:146 [INFO] found job 16912288 in provider astro proc_lsst.multi INFO: found job 16912288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:22 proc_lsst.multi:146 [INFO] found job 16912289 in provider astro proc_lsst.multi INFO: found job 16912289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:23 proc_lsst.multi:146 [INFO] found job 16912290 in provider astro proc_lsst.multi INFO: found job 16912290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:23 proc_lsst.multi:146 [INFO] found job 16912291 in provider astro proc_lsst.multi INFO: found job 16912291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:23 proc_lsst.multi:146 [INFO] found job 16912292 in provider astro proc_lsst.multi INFO: found job 16912292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:23 proc_lsst.multi:146 [INFO] found job 16912293 in provider astro proc_lsst.multi INFO: found job 16912293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:23 proc_lsst.multi:146 [INFO] found job 16912294 in provider astro proc_lsst.multi INFO: found job 16912294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:23 proc_lsst.multi:146 [INFO] found job 16912295 in provider astro proc_lsst.multi INFO: found job 16912295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:24 proc_lsst.multi:146 [INFO] found job 16912296 in provider astro proc_lsst.multi INFO: found job 16912296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:24 proc_lsst.multi:146 [INFO] found job 16912297 in provider astro proc_lsst.multi INFO: found job 16912297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:24 proc_lsst.multi:146 [INFO] found job 16912298 in provider astro proc_lsst.multi INFO: found job 16912298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:24 proc_lsst.multi:146 [INFO] found job 16912299 in provider astro proc_lsst.multi INFO: found job 16912299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:24 proc_lsst.multi:146 [INFO] found job 16912300 in provider astro proc_lsst.multi INFO: found job 16912300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:24 proc_lsst.multi:146 [INFO] found job 16912301 in provider astro proc_lsst.multi INFO: found job 16912301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:24 proc_lsst.multi:146 [INFO] found job 16912302 in provider astro proc_lsst.multi INFO: found job 16912302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:25 proc_lsst.multi:146 [INFO] found job 16912303 in provider astro proc_lsst.multi INFO: found job 16912303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:25 proc_lsst.multi:146 [INFO] found job 16912304 in provider astro proc_lsst.multi INFO: found job 16912304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:25 proc_lsst.multi:146 [INFO] found job 16912305 in provider astro proc_lsst.multi INFO: found job 16912305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:25 proc_lsst.multi:146 [INFO] found job 16912306 in provider astro proc_lsst.multi INFO: found job 16912306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:25 proc_lsst.multi:146 [INFO] found job 16912307 in provider astro proc_lsst.multi INFO: found job 16912307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:25 proc_lsst.multi:146 [INFO] found job 16912308 in provider astro proc_lsst.multi INFO: found job 16912308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:25 proc_lsst.multi:146 [INFO] found job 16912309 in provider astro proc_lsst.multi INFO: found job 16912309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:26 proc_lsst.multi:146 [INFO] found job 16912310 in provider astro proc_lsst.multi INFO: found job 16912310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:26 proc_lsst.multi:146 [INFO] found job 16912311 in provider astro proc_lsst.multi INFO: found job 16912311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:26 proc_lsst.multi:146 [INFO] found job 16912312 in provider astro proc_lsst.multi INFO: found job 16912312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912283,16912284,16912285,16912286,16912287,16912288,16912289,16912290,16912291,16912292,16912293,16912294,16912295,16912296,16912297,16912298,16912299,16912300,16912301,16912302,16912303,16912304,16912305,16912306,16912307,16912308,16912309,16912310,16912311,16912312' parsl.providers.slurm.slurm DEBUG: squeue returned 16912309 R 16912310 R 16912311 R 16912312 R 16912299 R 16912300 R 16912301 R 16912302 R 16912303 R 16912304 R 16912305 R 16912306 R 16912307 R 16912308 R 16912289 R 16912290 R 16912291 R 16912292 R 16912293 R 16912294 R 16912295 R 16912296 R 16912297 R 16912298 R 16912283 R 16912284 R 16912285 R 16912286 R 16912287 R 16912288 R parsl.providers.slurm.slurm DEBUG: Updating job 16912309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912283 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912284 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912285 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912286 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912288 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:26 proc_lsst.multi:146 [INFO] found job 16912313 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912313 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:26 proc_lsst.multi:146 [INFO] found job 16912314 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912314 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:26 proc_lsst.multi:146 [INFO] found job 16912315 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912315 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:26 proc_lsst.multi:146 [INFO] found job 16912316 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912316 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:27 proc_lsst.multi:146 [INFO] found job 16912317 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912313,16912314,16912315,16912316,16912317' parsl.providers.slurm.slurm DEBUG: squeue returned 16912313 R 16912314 R 16912315 R 16912316 R 16912317 R parsl.providers.slurm.slurm DEBUG: Updating job 16912313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912317 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:27 proc_lsst.multi:146 [INFO] found job 16912318 in provider escience_a40 proc_lsst.multi INFO: found job 16912318 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:27 proc_lsst.multi:146 [INFO] found job 16912319 in provider escience_a40 proc_lsst.multi INFO: found job 16912319 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:27 proc_lsst.multi:146 [INFO] found job 16912320 in provider escience_a40 proc_lsst.multi INFO: found job 16912320 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:27 proc_lsst.multi:146 [INFO] found job 16912321 in provider escience_a40 proc_lsst.multi INFO: found job 16912321 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:27 proc_lsst.multi:146 [INFO] found job 16912322 in provider escience_a40 proc_lsst.multi INFO: found job 16912322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:28 proc_lsst.multi:146 [INFO] found job 16912323 in provider escience_a40 proc_lsst.multi INFO: found job 16912323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:28 proc_lsst.multi:146 [INFO] found job 16912324 in provider escience_a40 proc_lsst.multi INFO: found job 16912324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:28 proc_lsst.multi:146 [INFO] found job 16912325 in provider escience_a40 proc_lsst.multi INFO: found job 16912325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:28 proc_lsst.multi:146 [INFO] found job 16912326 in provider escience_a40 proc_lsst.multi INFO: found job 16912326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:28 proc_lsst.multi:146 [INFO] found job 16912327 in provider escience_a40 proc_lsst.multi INFO: found job 16912327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:28 proc_lsst.multi:146 [INFO] found job 16912328 in provider escience_a40 proc_lsst.multi INFO: found job 16912328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:28 proc_lsst.multi:146 [INFO] found job 16912329 in provider escience_a40 proc_lsst.multi INFO: found job 16912329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:29 proc_lsst.multi:146 [INFO] found job 16912330 in provider escience_a40 proc_lsst.multi INFO: found job 16912330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:29 proc_lsst.multi:146 [INFO] found job 16912331 in provider escience_a40 proc_lsst.multi INFO: found job 16912331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:29 proc_lsst.multi:146 [INFO] found job 16912332 in provider escience_a40 proc_lsst.multi INFO: found job 16912332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:29 proc_lsst.multi:146 [INFO] found job 16912333 in provider escience_a40 proc_lsst.multi INFO: found job 16912333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:29 proc_lsst.multi:146 [INFO] found job 16912334 in provider escience_a40 proc_lsst.multi INFO: found job 16912334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:29 proc_lsst.multi:146 [INFO] found job 16912335 in provider escience_a40 proc_lsst.multi INFO: found job 16912335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:29 proc_lsst.multi:146 [INFO] found job 16912336 in provider escience_a40 proc_lsst.multi INFO: found job 16912336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:30 proc_lsst.multi:146 [INFO] found job 16912337 in provider escience_a40 proc_lsst.multi INFO: found job 16912337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912318,16912319,16912320,16912321,16912322,16912323,16912324,16912325,16912326,16912327,16912328,16912329,16912330,16912331,16912332,16912333,16912334,16912335,16912336,16912337' parsl.providers.slurm.slurm DEBUG: squeue returned 16912337 R 16912333 R 16912334 R 16912335 R 16912336 R 16912325 R 16912326 R 16912327 R 16912328 R 16912329 R 16912330 R 16912331 R 16912332 R 16912318 R 16912319 R 16912320 R 16912321 R 16912322 R 16912323 R 16912324 R parsl.providers.slurm.slurm DEBUG: Updating job 16912337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912321 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912324 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:30 proc_lsst.multi:146 [INFO] found job 16912338 in provider checkpoint proc_lsst.multi INFO: found job 16912338 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:30 proc_lsst.multi:146 [INFO] found job 16912339 in provider checkpoint proc_lsst.multi INFO: found job 16912339 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:30 proc_lsst.multi:146 [INFO] found job 16912340 in provider checkpoint proc_lsst.multi INFO: found job 16912340 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:30 proc_lsst.multi:146 [INFO] found job 16912341 in provider checkpoint proc_lsst.multi INFO: found job 16912341 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:30 proc_lsst.multi:146 [INFO] found job 16912342 in provider checkpoint proc_lsst.multi INFO: found job 16912342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:30 proc_lsst.multi:146 [INFO] found job 16912343 in provider checkpoint proc_lsst.multi INFO: found job 16912343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:31 proc_lsst.multi:146 [INFO] found job 16912344 in provider checkpoint proc_lsst.multi INFO: found job 16912344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:31 proc_lsst.multi:146 [INFO] found job 16912345 in provider checkpoint proc_lsst.multi INFO: found job 16912345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:31 proc_lsst.multi:146 [INFO] found job 16912346 in provider checkpoint proc_lsst.multi INFO: found job 16912346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:31 proc_lsst.multi:146 [INFO] found job 16912347 in provider checkpoint proc_lsst.multi INFO: found job 16912347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:31 proc_lsst.multi:146 [INFO] found job 16912348 in provider checkpoint proc_lsst.multi INFO: found job 16912348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:31 proc_lsst.multi:146 [INFO] found job 16912349 in provider checkpoint proc_lsst.multi INFO: found job 16912349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:31 proc_lsst.multi:146 [INFO] found job 16912350 in provider checkpoint proc_lsst.multi INFO: found job 16912350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:32 proc_lsst.multi:146 [INFO] found job 16912351 in provider checkpoint proc_lsst.multi INFO: found job 16912351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:32 proc_lsst.multi:146 [INFO] found job 16912352 in provider checkpoint proc_lsst.multi INFO: found job 16912352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:32 proc_lsst.multi:146 [INFO] found job 16912353 in provider checkpoint proc_lsst.multi INFO: found job 16912353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:32 proc_lsst.multi:146 [INFO] found job 16912354 in provider checkpoint proc_lsst.multi INFO: found job 16912354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:32 proc_lsst.multi:146 [INFO] found job 16912355 in provider checkpoint proc_lsst.multi INFO: found job 16912355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:33 proc_lsst.multi:146 [INFO] found job 16912356 in provider checkpoint proc_lsst.multi INFO: found job 16912356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:33 proc_lsst.multi:146 [INFO] found job 16912359 in provider checkpoint proc_lsst.multi INFO: found job 16912359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:33 proc_lsst.multi:146 [INFO] found job 16912360 in provider checkpoint proc_lsst.multi INFO: found job 16912360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:33 proc_lsst.multi:146 [INFO] found job 16912361 in provider checkpoint proc_lsst.multi INFO: found job 16912361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:33 proc_lsst.multi:146 [INFO] found job 16912362 in provider checkpoint proc_lsst.multi INFO: found job 16912362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:33 proc_lsst.multi:146 [INFO] found job 16912363 in provider checkpoint proc_lsst.multi INFO: found job 16912363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:34 proc_lsst.multi:146 [INFO] found job 16912364 in provider checkpoint proc_lsst.multi INFO: found job 16912364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:34 proc_lsst.multi:146 [INFO] found job 16912365 in provider checkpoint proc_lsst.multi INFO: found job 16912365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:34 proc_lsst.multi:146 [INFO] found job 16912366 in provider checkpoint proc_lsst.multi INFO: found job 16912366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:34 proc_lsst.multi:146 [INFO] found job 16912367 in provider checkpoint proc_lsst.multi INFO: found job 16912367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:34 proc_lsst.multi:146 [INFO] found job 16912368 in provider checkpoint proc_lsst.multi INFO: found job 16912368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:34 proc_lsst.multi:146 [INFO] found job 16912369 in provider checkpoint proc_lsst.multi INFO: found job 16912369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:34 proc_lsst.multi:146 [INFO] found job 16912370 in provider checkpoint proc_lsst.multi INFO: found job 16912370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:35 proc_lsst.multi:146 [INFO] found job 16912371 in provider checkpoint proc_lsst.multi INFO: found job 16912371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:35 proc_lsst.multi:146 [INFO] found job 16912372 in provider checkpoint proc_lsst.multi INFO: found job 16912372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:35 proc_lsst.multi:146 [INFO] found job 16912373 in provider checkpoint proc_lsst.multi INFO: found job 16912373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:35 proc_lsst.multi:146 [INFO] found job 16912374 in provider checkpoint proc_lsst.multi INFO: found job 16912374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:35 proc_lsst.multi:146 [INFO] found job 16912375 in provider checkpoint proc_lsst.multi INFO: found job 16912375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING 2024-03-08 07:29:35 proc_lsst.multi:146 [INFO] found job 16912378 in provider checkpoint proc_lsst.multi INFO: found job 16912378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16912338,16912339,16912340,16912341,16912342,16912343,16912344,16912345,16912346,16912347,16912348,16912349,16912350,16912351,16912352,16912353,16912354,16912355,16912356,16912359,16912360,16912361,16912362,16912363,16912364,16912365,16912366,16912367,16912368,16912369,16912370,16912371,16912372,16912373,16912374,16912375,16912378' parsl.providers.slurm.slurm DEBUG: squeue returned 16912378 R 16912363 R 16912364 R 16912365 R 16912366 R 16912367 R 16912368 R 16912369 R 16912370 R 16912371 R 16912372 R 16912373 R 16912374 R 16912375 R 16912355 R 16912356 R 16912359 R 16912360 R 16912361 R 16912362 R 16912338 R 16912339 R 16912340 R 16912341 R 16912342 R 16912343 R 16912344 R 16912345 R 16912346 R 16912347 R 16912348 R 16912349 R 16912350 R 16912351 R 16912352 R 16912353 R 16912354 R parsl.providers.slurm.slurm DEBUG: Updating job 16912378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16912354 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 361 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22721853032400 parsl.dataflow.dflow INFO: Task 866 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0828200d4e611cd4767be34a9aa64bb with result from task 866 parsl.dataflow.dflow INFO: DFK cleanup initiated parsl.dataflow.dflow INFO: Summary of tasks in DFK: parsl.dataflow.dflow INFO: Tasks in state States.unsched: 0 parsl.dataflow.dflow INFO: Tasks in state States.pending: 0 parsl.dataflow.dflow DEBUG: Task 866 checkpointed parsl.dataflow.dflow INFO: Tasks in state States.running: 0 parsl.dataflow.dflow INFO: Tasks in state States.exec_done: 1104 parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Tasks in state States.failed: 6 parsl.dataflow.dflow INFO: Standard output for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/638505f0-3245-4c61-9d1e-6e363b15b263_calibrate_855136_61.stdout parsl.dataflow.dflow INFO: Tasks in state States.dep_fail: 6 parsl.dataflow.dflow INFO: Standard error for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/submit/DEEP/20190504/A0b/science#step1/20240308T071422Z/logs/calibrate/855136/638505f0-3245-4c61-9d1e-6e363b15b263_calibrate_855136_61.stderr parsl.dataflow.dflow INFO: Tasks in state States.launched: 0 parsl.dataflow.dflow INFO: Tasks in state States.fail_retryable: 0 parsl.dataflow.dflow INFO: Tasks in state States.memo_done: 0 parsl.dataflow.dflow INFO: Tasks in state States.joining: 0 parsl.dataflow.dflow INFO: Tasks in state States.running_ended: 0 parsl.dataflow.dflow INFO: End of summary parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Closing job status poller parsl.dataflow.dflow INFO: Terminated job status poller parsl.dataflow.dflow INFO: Scaling in and shutting down executors parsl.dataflow.dflow INFO: Scaling in executor multi parsl.executors.high_throughput.executor DEBUG: Scale in called, blocks=93, block_ids=[] parsl.executors.high_throughput.executor DEBUG: Scale in selecting from 94 blocks parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a3cc9f7667aa parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a3cc9f7667aa parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a454e7881b70 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a454e7881b70 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6d057e8985cd parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6d057e8985cd parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 52c5a30126a2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 52c5a30126a2 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ac46c33d1561 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ac46c33d1561 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bdd2f8e8f67b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bdd2f8e8f67b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a78b00546de1 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a78b00546de1 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 787734c3eccd parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 787734c3eccd parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b24e5d3383e6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b24e5d3383e6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: efc3252fdecf parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: efc3252fdecf parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: aa3fb00aeb8f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: aa3fb00aeb8f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ccdd7ccbcb6f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ccdd7ccbcb6f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6b5092ba0d5b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6b5092ba0d5b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8ef142a4c3b9 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8ef142a4c3b9 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f61aab7ba509 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f61aab7ba509 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1ecff319aff5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1ecff319aff5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 82a7e7f65b20 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 82a7e7f65b20 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: de1d57ef6dd3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: de1d57ef6dd3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b48dd9f4a89b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b48dd9f4a89b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: dd7914955c2f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: dd7914955c2f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4e9bf3bc4af9 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4e9bf3bc4af9 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e870e9fd9e68 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e870e9fd9e68 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ef0797387431 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ef0797387431 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bf55c2117cad parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bf55c2117cad parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c18a528503fc parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c18a528503fc parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8a27e199b461 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8a27e199b461 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c4286ff19916 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c4286ff19916 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7a3a8515daa4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7a3a8515daa4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5e613c391546 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5e613c391546 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a0836fd7be45 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a0836fd7be45 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b84e82a66902 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b84e82a66902 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8d1f5b1bdecf parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8d1f5b1bdecf parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 65e5aff89f82 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 65e5aff89f82 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 615d2ce47baf parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 615d2ce47baf parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7a24f39eab81 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7a24f39eab81 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6007b4e4612c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6007b4e4612c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0353f2542f1a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0353f2542f1a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 822d4b1536e9 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 822d4b1536e9 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 05cfb036d789 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 05cfb036d789 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d19addf72a39 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d19addf72a39 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5e3080a08c14 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5e3080a08c14 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 419de959a9e3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 419de959a9e3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1ee9910bb73a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1ee9910bb73a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a3c28f265529 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a3c28f265529 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4b16f7ea48e5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4b16f7ea48e5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6b30d06b730d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6b30d06b730d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8e5ce4af5340 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8e5ce4af5340 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 025d8bde6a09 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 025d8bde6a09 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1918b82bd191 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1918b82bd191 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9ff849688f31 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9ff849688f31 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ba4bc986903e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ba4bc986903e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b1064cbd6a4d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b1064cbd6a4d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 270062ba5cda parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 270062ba5cda parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1d5c294a0716 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1d5c294a0716 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1e2811b892c9 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1e2811b892c9 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 329c9be4c1fb parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 329c9be4c1fb parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 89e258ef72c0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 89e258ef72c0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 26b44818b43b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 26b44818b43b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 73a3ce4fa874 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 73a3ce4fa874 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: adc1d57da472 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: adc1d57da472 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1df2b0b36374 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1df2b0b36374 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 263733116a67 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 263733116a67 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fa57b3f81e3b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fa57b3f81e3b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0ac2887cf748 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0ac2887cf748 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 32e7dcf955c7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 32e7dcf955c7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 535d0e11c259 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 535d0e11c259 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: aec8f750608d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: aec8f750608d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 68be55487b0e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 68be55487b0e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3feaccaf77b4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3feaccaf77b4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7e5487922edc parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7e5487922edc parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bebd612d152b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bebd612d152b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7cf453268ca7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7cf453268ca7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b5ef2803b6f1 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b5ef2803b6f1 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8351722906af parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8351722906af parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f4d194cdc214 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f4d194cdc214 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 07cc76698876 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 07cc76698876 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 41a519b5e4b5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 41a519b5e4b5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4c7bcf23dc47 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4c7bcf23dc47 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d2956ab66b62 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d2956ab66b62 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 84f83d0200c7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 84f83d0200c7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1b97deafa41a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1b97deafa41a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0fe5d8dbef0d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0fe5d8dbef0d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c8769506281b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c8769506281b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9c21fe49a406 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9c21fe49a406 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e83b7ade41fb parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e83b7ade41fb parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 439356214416 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 439356214416 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6ab99d2d9140 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6ab99d2d9140 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a9353a1ea8bf parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a9353a1ea8bf parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8cb423e0a410 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8cb423e0a410 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f3689e9ffaad parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f3689e9ffaad parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6b38d97bfe13 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6b38d97bfe13 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ee8192af5cc0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ee8192af5cc0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 73a40dabd587 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 73a40dabd587 2024-03-08 07:29:42 proc_lsst.multi:146 [INFO] found job 16912340 in provider checkpoint proc_lsst.multi INFO: found job 16912340 in provider checkpoint 2024-03-08 07:29:42 proc_lsst.multi:201 [INFO] cancelling 16912340 on provider checkpoint proc_lsst.multi INFO: cancelling 16912340 on provider checkpoint 2024-03-08 07:29:42 proc_lsst.multi:146 [INFO] found job 16912367 in provider checkpoint proc_lsst.multi INFO: found job 16912367 in provider checkpoint 2024-03-08 07:29:42 proc_lsst.multi:201 [INFO] cancelling 16912367 on provider checkpoint proc_lsst.multi INFO: cancelling 16912367 on provider checkpoint 2024-03-08 07:29:42 proc_lsst.multi:146 [INFO] found job 16912300 in provider astro proc_lsst.multi INFO: found job 16912300 in provider astro 2024-03-08 07:29:42 proc_lsst.multi:201 [INFO] cancelling 16912300 on provider astro proc_lsst.multi INFO: cancelling 16912300 on provider astro 2024-03-08 07:29:42 proc_lsst.multi:146 [INFO] found job 16912372 in provider checkpoint proc_lsst.multi INFO: found job 16912372 in provider checkpoint 2024-03-08 07:29:42 proc_lsst.multi:201 [INFO] cancelling 16912372 on provider checkpoint proc_lsst.multi INFO: cancelling 16912372 on provider checkpoint 2024-03-08 07:29:42 proc_lsst.multi:146 [INFO] found job 16912328 in provider escience_a40 proc_lsst.multi INFO: found job 16912328 in provider escience_a40 2024-03-08 07:29:42 proc_lsst.multi:201 [INFO] cancelling 16912328 on provider escience_a40 proc_lsst.multi INFO: cancelling 16912328 on provider escience_a40 2024-03-08 07:29:42 proc_lsst.multi:146 [INFO] found job 16912341 in provider checkpoint proc_lsst.multi INFO: found job 16912341 in provider checkpoint 2024-03-08 07:29:42 proc_lsst.multi:201 [INFO] cancelling 16912341 on provider checkpoint proc_lsst.multi INFO: cancelling 16912341 on provider checkpoint 2024-03-08 07:29:42 proc_lsst.multi:146 [INFO] found job 16912335 in provider escience_a40 proc_lsst.multi INFO: found job 16912335 in provider escience_a40 2024-03-08 07:29:42 proc_lsst.multi:201 [INFO] cancelling 16912335 on provider escience_a40 proc_lsst.multi INFO: cancelling 16912335 on provider escience_a40 2024-03-08 07:29:42 proc_lsst.multi:146 [INFO] found job 16912297 in provider astro proc_lsst.multi INFO: found job 16912297 in provider astro 2024-03-08 07:29:42 proc_lsst.multi:201 [INFO] cancelling 16912297 on provider astro proc_lsst.multi INFO: cancelling 16912297 on provider astro 2024-03-08 07:29:42 proc_lsst.multi:146 [INFO] found job 16912284 in provider astro proc_lsst.multi INFO: found job 16912284 in provider astro 2024-03-08 07:29:42 proc_lsst.multi:201 [INFO] cancelling 16912284 on provider astro proc_lsst.multi INFO: cancelling 16912284 on provider astro 2024-03-08 07:29:43 proc_lsst.multi:146 [INFO] found job 16912362 in provider checkpoint proc_lsst.multi INFO: found job 16912362 in provider checkpoint 2024-03-08 07:29:43 proc_lsst.multi:201 [INFO] cancelling 16912362 on provider checkpoint proc_lsst.multi INFO: cancelling 16912362 on provider checkpoint 2024-03-08 07:29:43 proc_lsst.multi:146 [INFO] found job 16912354 in provider checkpoint proc_lsst.multi INFO: found job 16912354 in provider checkpoint 2024-03-08 07:29:43 proc_lsst.multi:201 [INFO] cancelling 16912354 on provider checkpoint proc_lsst.multi INFO: cancelling 16912354 on provider checkpoint 2024-03-08 07:29:43 proc_lsst.multi:146 [INFO] found job 16912303 in provider astro proc_lsst.multi INFO: found job 16912303 in provider astro 2024-03-08 07:29:43 proc_lsst.multi:201 [INFO] cancelling 16912303 on provider astro proc_lsst.multi INFO: cancelling 16912303 on provider astro 2024-03-08 07:29:43 proc_lsst.multi:146 [INFO] found job 16912339 in provider checkpoint proc_lsst.multi INFO: found job 16912339 in provider checkpoint 2024-03-08 07:29:43 proc_lsst.multi:201 [INFO] cancelling 16912339 on provider checkpoint proc_lsst.multi INFO: cancelling 16912339 on provider checkpoint 2024-03-08 07:29:43 proc_lsst.multi:146 [INFO] found job 16912347 in provider checkpoint proc_lsst.multi INFO: found job 16912347 in provider checkpoint 2024-03-08 07:29:43 proc_lsst.multi:201 [INFO] cancelling 16912347 on provider checkpoint proc_lsst.multi INFO: cancelling 16912347 on provider checkpoint 2024-03-08 07:29:43 proc_lsst.multi:146 [INFO] found job 16912368 in provider checkpoint proc_lsst.multi INFO: found job 16912368 in provider checkpoint 2024-03-08 07:29:43 proc_lsst.multi:201 [INFO] cancelling 16912368 on provider checkpoint proc_lsst.multi INFO: cancelling 16912368 on provider checkpoint 2024-03-08 07:29:43 proc_lsst.multi:146 [INFO] found job 16912355 in provider checkpoint proc_lsst.multi INFO: found job 16912355 in provider checkpoint 2024-03-08 07:29:43 proc_lsst.multi:201 [INFO] cancelling 16912355 on provider checkpoint proc_lsst.multi INFO: cancelling 16912355 on provider checkpoint 2024-03-08 07:29:43 proc_lsst.multi:146 [INFO] found job 16912342 in provider checkpoint proc_lsst.multi INFO: found job 16912342 in provider checkpoint 2024-03-08 07:29:43 proc_lsst.multi:201 [INFO] cancelling 16912342 on provider checkpoint proc_lsst.multi INFO: cancelling 16912342 on provider checkpoint 2024-03-08 07:29:44 proc_lsst.multi:146 [INFO] found job 16912374 in provider checkpoint proc_lsst.multi INFO: found job 16912374 in provider checkpoint 2024-03-08 07:29:44 proc_lsst.multi:201 [INFO] cancelling 16912374 on provider checkpoint proc_lsst.multi INFO: cancelling 16912374 on provider checkpoint 2024-03-08 07:29:44 proc_lsst.multi:146 [INFO] found job 16912365 in provider checkpoint proc_lsst.multi INFO: found job 16912365 in provider checkpoint 2024-03-08 07:29:44 proc_lsst.multi:201 [INFO] cancelling 16912365 on provider checkpoint proc_lsst.multi INFO: cancelling 16912365 on provider checkpoint 2024-03-08 07:29:44 proc_lsst.multi:146 [INFO] found job 16912349 in provider checkpoint proc_lsst.multi INFO: found job 16912349 in provider checkpoint 2024-03-08 07:29:44 proc_lsst.multi:201 [INFO] cancelling 16912349 on provider checkpoint proc_lsst.multi INFO: cancelling 16912349 on provider checkpoint 2024-03-08 07:29:44 proc_lsst.multi:146 [INFO] found job 16912361 in provider checkpoint proc_lsst.multi INFO: found job 16912361 in provider checkpoint 2024-03-08 07:29:44 proc_lsst.multi:201 [INFO] cancelling 16912361 on provider checkpoint proc_lsst.multi INFO: cancelling 16912361 on provider checkpoint 2024-03-08 07:29:44 proc_lsst.multi:146 [INFO] found job 16912346 in provider checkpoint proc_lsst.multi INFO: found job 16912346 in provider checkpoint 2024-03-08 07:29:44 proc_lsst.multi:201 [INFO] cancelling 16912346 on provider checkpoint proc_lsst.multi INFO: cancelling 16912346 on provider checkpoint 2024-03-08 07:29:44 proc_lsst.multi:146 [INFO] found job 16912363 in provider checkpoint proc_lsst.multi INFO: found job 16912363 in provider checkpoint 2024-03-08 07:29:44 proc_lsst.multi:201 [INFO] cancelling 16912363 on provider checkpoint proc_lsst.multi INFO: cancelling 16912363 on provider checkpoint 2024-03-08 07:29:44 proc_lsst.multi:146 [INFO] found job 16912345 in provider checkpoint proc_lsst.multi INFO: found job 16912345 in provider checkpoint 2024-03-08 07:29:44 proc_lsst.multi:201 [INFO] cancelling 16912345 on provider checkpoint proc_lsst.multi INFO: cancelling 16912345 on provider checkpoint 2024-03-08 07:29:44 proc_lsst.multi:146 [INFO] found job 16912321 in provider escience_a40 proc_lsst.multi INFO: found job 16912321 in provider escience_a40 2024-03-08 07:29:44 proc_lsst.multi:201 [INFO] cancelling 16912321 on provider escience_a40 proc_lsst.multi INFO: cancelling 16912321 on provider escience_a40 2024-03-08 07:29:44 proc_lsst.multi:146 [INFO] found job 16912366 in provider checkpoint proc_lsst.multi INFO: found job 16912366 in provider checkpoint 2024-03-08 07:29:44 proc_lsst.multi:201 [INFO] cancelling 16912366 on provider checkpoint proc_lsst.multi INFO: cancelling 16912366 on provider checkpoint 2024-03-08 07:29:44 proc_lsst.multi:146 [INFO] found job 16912337 in provider escience_a40 proc_lsst.multi INFO: found job 16912337 in provider escience_a40 2024-03-08 07:29:44 proc_lsst.multi:201 [INFO] cancelling 16912337 on provider escience_a40 proc_lsst.multi INFO: cancelling 16912337 on provider escience_a40 2024-03-08 07:29:45 proc_lsst.multi:146 [INFO] found job 16912364 in provider checkpoint proc_lsst.multi INFO: found job 16912364 in provider checkpoint 2024-03-08 07:29:45 proc_lsst.multi:201 [INFO] cancelling 16912364 on provider checkpoint proc_lsst.multi INFO: cancelling 16912364 on provider checkpoint 2024-03-08 07:29:45 proc_lsst.multi:146 [INFO] found job 16912334 in provider escience_a40 proc_lsst.multi INFO: found job 16912334 in provider escience_a40 2024-03-08 07:29:45 proc_lsst.multi:201 [INFO] cancelling 16912334 on provider escience_a40 proc_lsst.multi INFO: cancelling 16912334 on provider escience_a40 2024-03-08 07:29:45 proc_lsst.multi:146 [INFO] found job 16912315 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912315 in provider escience_rtx6k 2024-03-08 07:29:45 proc_lsst.multi:201 [INFO] cancelling 16912315 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16912315 on provider escience_rtx6k 2024-03-08 07:29:45 proc_lsst.multi:146 [INFO] found job 16912287 in provider astro proc_lsst.multi INFO: found job 16912287 in provider astro 2024-03-08 07:29:45 proc_lsst.multi:201 [INFO] cancelling 16912287 on provider astro proc_lsst.multi INFO: cancelling 16912287 on provider astro 2024-03-08 07:29:45 proc_lsst.multi:146 [INFO] found job 16912350 in provider checkpoint proc_lsst.multi INFO: found job 16912350 in provider checkpoint 2024-03-08 07:29:45 proc_lsst.multi:201 [INFO] cancelling 16912350 on provider checkpoint proc_lsst.multi INFO: cancelling 16912350 on provider checkpoint 2024-03-08 07:29:45 proc_lsst.multi:146 [INFO] found job 16912336 in provider escience_a40 proc_lsst.multi INFO: found job 16912336 in provider escience_a40 2024-03-08 07:29:45 proc_lsst.multi:201 [INFO] cancelling 16912336 on provider escience_a40 proc_lsst.multi INFO: cancelling 16912336 on provider escience_a40 2024-03-08 07:29:45 proc_lsst.multi:146 [INFO] found job 16912373 in provider checkpoint proc_lsst.multi INFO: found job 16912373 in provider checkpoint 2024-03-08 07:29:45 proc_lsst.multi:201 [INFO] cancelling 16912373 on provider checkpoint proc_lsst.multi INFO: cancelling 16912373 on provider checkpoint 2024-03-08 07:29:45 proc_lsst.multi:146 [INFO] found job 16912378 in provider checkpoint proc_lsst.multi INFO: found job 16912378 in provider checkpoint 2024-03-08 07:29:45 proc_lsst.multi:201 [INFO] cancelling 16912378 on provider checkpoint proc_lsst.multi INFO: cancelling 16912378 on provider checkpoint 2024-03-08 07:29:45 proc_lsst.multi:146 [INFO] found job 16912319 in provider escience_a40 proc_lsst.multi INFO: found job 16912319 in provider escience_a40 2024-03-08 07:29:45 proc_lsst.multi:201 [INFO] cancelling 16912319 on provider escience_a40 proc_lsst.multi INFO: cancelling 16912319 on provider escience_a40 2024-03-08 07:29:46 proc_lsst.multi:146 [INFO] found job 16912351 in provider checkpoint proc_lsst.multi INFO: found job 16912351 in provider checkpoint 2024-03-08 07:29:46 proc_lsst.multi:201 [INFO] cancelling 16912351 on provider checkpoint proc_lsst.multi INFO: cancelling 16912351 on provider checkpoint 2024-03-08 07:29:46 proc_lsst.multi:146 [INFO] found job 16912348 in provider checkpoint proc_lsst.multi INFO: found job 16912348 in provider checkpoint 2024-03-08 07:29:46 proc_lsst.multi:201 [INFO] cancelling 16912348 on provider checkpoint proc_lsst.multi INFO: cancelling 16912348 on provider checkpoint 2024-03-08 07:29:46 proc_lsst.multi:146 [INFO] found job 16912326 in provider escience_a40 proc_lsst.multi INFO: found job 16912326 in provider escience_a40 2024-03-08 07:29:46 proc_lsst.multi:201 [INFO] cancelling 16912326 on provider escience_a40 proc_lsst.multi INFO: cancelling 16912326 on provider escience_a40 2024-03-08 07:29:46 proc_lsst.multi:146 [INFO] found job 16912318 in provider escience_a40 proc_lsst.multi INFO: found job 16912318 in provider escience_a40 2024-03-08 07:29:46 proc_lsst.multi:201 [INFO] cancelling 16912318 on provider escience_a40 proc_lsst.multi INFO: cancelling 16912318 on provider escience_a40 2024-03-08 07:29:46 proc_lsst.multi:146 [INFO] found job 16912370 in provider checkpoint proc_lsst.multi INFO: found job 16912370 in provider checkpoint 2024-03-08 07:29:46 proc_lsst.multi:201 [INFO] cancelling 16912370 on provider checkpoint proc_lsst.multi INFO: cancelling 16912370 on provider checkpoint 2024-03-08 07:29:46 proc_lsst.multi:146 [INFO] found job 16912310 in provider astro proc_lsst.multi INFO: found job 16912310 in provider astro 2024-03-08 07:29:46 proc_lsst.multi:201 [INFO] cancelling 16912310 on provider astro proc_lsst.multi INFO: cancelling 16912310 on provider astro 2024-03-08 07:29:46 proc_lsst.multi:146 [INFO] found job 16912302 in provider astro proc_lsst.multi INFO: found job 16912302 in provider astro 2024-03-08 07:29:46 proc_lsst.multi:201 [INFO] cancelling 16912302 on provider astro proc_lsst.multi INFO: cancelling 16912302 on provider astro 2024-03-08 07:29:46 proc_lsst.multi:146 [INFO] found job 16912323 in provider escience_a40 proc_lsst.multi INFO: found job 16912323 in provider escience_a40 2024-03-08 07:29:46 proc_lsst.multi:201 [INFO] cancelling 16912323 on provider escience_a40 proc_lsst.multi INFO: cancelling 16912323 on provider escience_a40 2024-03-08 07:29:46 proc_lsst.multi:146 [INFO] found job 16912304 in provider astro proc_lsst.multi INFO: found job 16912304 in provider astro 2024-03-08 07:29:46 proc_lsst.multi:201 [INFO] cancelling 16912304 on provider astro proc_lsst.multi INFO: cancelling 16912304 on provider astro 2024-03-08 07:29:47 proc_lsst.multi:146 [INFO] found job 16912352 in provider checkpoint proc_lsst.multi INFO: found job 16912352 in provider checkpoint 2024-03-08 07:29:47 proc_lsst.multi:201 [INFO] cancelling 16912352 on provider checkpoint proc_lsst.multi INFO: cancelling 16912352 on provider checkpoint 2024-03-08 07:29:47 proc_lsst.multi:146 [INFO] found job 16912288 in provider astro proc_lsst.multi INFO: found job 16912288 in provider astro 2024-03-08 07:29:47 proc_lsst.multi:201 [INFO] cancelling 16912288 on provider astro proc_lsst.multi INFO: cancelling 16912288 on provider astro 2024-03-08 07:29:47 proc_lsst.multi:146 [INFO] found job 16912320 in provider escience_a40 proc_lsst.multi INFO: found job 16912320 in provider escience_a40 2024-03-08 07:29:47 proc_lsst.multi:201 [INFO] cancelling 16912320 on provider escience_a40 proc_lsst.multi INFO: cancelling 16912320 on provider escience_a40 2024-03-08 07:29:47 proc_lsst.multi:146 [INFO] found job 75775 in provider local proc_lsst.multi INFO: found job 75775 in provider local 2024-03-08 07:29:47 proc_lsst.multi:201 [INFO] cancelling 75775 on provider local proc_lsst.multi INFO: cancelling 75775 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 75775 2024-03-08 07:29:47 proc_lsst.multi:146 [INFO] found job 16912305 in provider astro proc_lsst.multi INFO: found job 16912305 in provider astro 2024-03-08 07:29:47 proc_lsst.multi:201 [INFO] cancelling 16912305 on provider astro proc_lsst.multi INFO: cancelling 16912305 on provider astro 2024-03-08 07:29:47 proc_lsst.multi:146 [INFO] found job 16912353 in provider checkpoint proc_lsst.multi INFO: found job 16912353 in provider checkpoint 2024-03-08 07:29:47 proc_lsst.multi:201 [INFO] cancelling 16912353 on provider checkpoint proc_lsst.multi INFO: cancelling 16912353 on provider checkpoint 2024-03-08 07:29:47 proc_lsst.multi:146 [INFO] found job 16912293 in provider astro proc_lsst.multi INFO: found job 16912293 in provider astro 2024-03-08 07:29:47 proc_lsst.multi:201 [INFO] cancelling 16912293 on provider astro proc_lsst.multi INFO: cancelling 16912293 on provider astro 2024-03-08 07:29:47 proc_lsst.multi:146 [INFO] found job 16912301 in provider astro proc_lsst.multi INFO: found job 16912301 in provider astro 2024-03-08 07:29:47 proc_lsst.multi:201 [INFO] cancelling 16912301 on provider astro proc_lsst.multi INFO: cancelling 16912301 on provider astro 2024-03-08 07:29:48 proc_lsst.multi:146 [INFO] found job 16912338 in provider checkpoint proc_lsst.multi INFO: found job 16912338 in provider checkpoint 2024-03-08 07:29:48 proc_lsst.multi:201 [INFO] cancelling 16912338 on provider checkpoint proc_lsst.multi INFO: cancelling 16912338 on provider checkpoint 2024-03-08 07:29:48 proc_lsst.multi:146 [INFO] found job 16912360 in provider checkpoint proc_lsst.multi INFO: found job 16912360 in provider checkpoint 2024-03-08 07:29:48 proc_lsst.multi:201 [INFO] cancelling 16912360 on provider checkpoint proc_lsst.multi INFO: cancelling 16912360 on provider checkpoint 2024-03-08 07:29:48 proc_lsst.multi:146 [INFO] found job 16912369 in provider checkpoint proc_lsst.multi INFO: found job 16912369 in provider checkpoint 2024-03-08 07:29:48 proc_lsst.multi:201 [INFO] cancelling 16912369 on provider checkpoint proc_lsst.multi INFO: cancelling 16912369 on provider checkpoint 2024-03-08 07:29:48 proc_lsst.multi:146 [INFO] found job 16912332 in provider escience_a40 proc_lsst.multi INFO: found job 16912332 in provider escience_a40 2024-03-08 07:29:48 proc_lsst.multi:201 [INFO] cancelling 16912332 on provider escience_a40 proc_lsst.multi INFO: cancelling 16912332 on provider escience_a40 2024-03-08 07:29:48 proc_lsst.multi:146 [INFO] found job 16912294 in provider astro proc_lsst.multi INFO: found job 16912294 in provider astro 2024-03-08 07:29:48 proc_lsst.multi:201 [INFO] cancelling 16912294 on provider astro proc_lsst.multi INFO: cancelling 16912294 on provider astro 2024-03-08 07:29:48 proc_lsst.multi:146 [INFO] found job 16912289 in provider astro proc_lsst.multi INFO: found job 16912289 in provider astro 2024-03-08 07:29:48 proc_lsst.multi:201 [INFO] cancelling 16912289 on provider astro proc_lsst.multi INFO: cancelling 16912289 on provider astro 2024-03-08 07:29:48 proc_lsst.multi:146 [INFO] found job 16912327 in provider escience_a40 proc_lsst.multi INFO: found job 16912327 in provider escience_a40 2024-03-08 07:29:48 proc_lsst.multi:201 [INFO] cancelling 16912327 on provider escience_a40 proc_lsst.multi INFO: cancelling 16912327 on provider escience_a40 2024-03-08 07:29:48 proc_lsst.multi:146 [INFO] found job 16912317 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912317 in provider escience_rtx6k 2024-03-08 07:29:48 proc_lsst.multi:201 [INFO] cancelling 16912317 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16912317 on provider escience_rtx6k 2024-03-08 07:29:49 proc_lsst.multi:146 [INFO] found job 16912313 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912313 in provider escience_rtx6k 2024-03-08 07:29:49 proc_lsst.multi:201 [INFO] cancelling 16912313 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16912313 on provider escience_rtx6k 2024-03-08 07:29:49 proc_lsst.multi:146 [INFO] found job 16912295 in provider astro proc_lsst.multi INFO: found job 16912295 in provider astro 2024-03-08 07:29:49 proc_lsst.multi:201 [INFO] cancelling 16912295 on provider astro proc_lsst.multi INFO: cancelling 16912295 on provider astro 2024-03-08 07:29:49 proc_lsst.multi:146 [INFO] found job 16912308 in provider astro proc_lsst.multi INFO: found job 16912308 in provider astro 2024-03-08 07:29:49 proc_lsst.multi:201 [INFO] cancelling 16912308 on provider astro proc_lsst.multi INFO: cancelling 16912308 on provider astro 2024-03-08 07:29:49 proc_lsst.multi:146 [INFO] found job 16912316 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912316 in provider escience_rtx6k 2024-03-08 07:29:49 proc_lsst.multi:201 [INFO] cancelling 16912316 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16912316 on provider escience_rtx6k 2024-03-08 07:29:49 proc_lsst.multi:146 [INFO] found job 16912356 in provider checkpoint proc_lsst.multi INFO: found job 16912356 in provider checkpoint 2024-03-08 07:29:49 proc_lsst.multi:201 [INFO] cancelling 16912356 on provider checkpoint proc_lsst.multi INFO: cancelling 16912356 on provider checkpoint 2024-03-08 07:29:49 proc_lsst.multi:146 [INFO] found job 16912283 in provider astro proc_lsst.multi INFO: found job 16912283 in provider astro 2024-03-08 07:29:49 proc_lsst.multi:201 [INFO] cancelling 16912283 on provider astro proc_lsst.multi INFO: cancelling 16912283 on provider astro 2024-03-08 07:29:49 proc_lsst.multi:146 [INFO] found job 16912359 in provider checkpoint proc_lsst.multi INFO: found job 16912359 in provider checkpoint 2024-03-08 07:29:49 proc_lsst.multi:201 [INFO] cancelling 16912359 on provider checkpoint proc_lsst.multi INFO: cancelling 16912359 on provider checkpoint 2024-03-08 07:29:49 proc_lsst.multi:146 [INFO] found job 16912343 in provider checkpoint proc_lsst.multi INFO: found job 16912343 in provider checkpoint 2024-03-08 07:29:49 proc_lsst.multi:201 [INFO] cancelling 16912343 on provider checkpoint proc_lsst.multi INFO: cancelling 16912343 on provider checkpoint 2024-03-08 07:29:50 proc_lsst.multi:146 [INFO] found job 16912311 in provider astro proc_lsst.multi INFO: found job 16912311 in provider astro 2024-03-08 07:29:50 proc_lsst.multi:201 [INFO] cancelling 16912311 on provider astro proc_lsst.multi INFO: cancelling 16912311 on provider astro 2024-03-08 07:29:50 proc_lsst.multi:146 [INFO] found job 16912307 in provider astro proc_lsst.multi INFO: found job 16912307 in provider astro 2024-03-08 07:29:50 proc_lsst.multi:201 [INFO] cancelling 16912307 on provider astro proc_lsst.multi INFO: cancelling 16912307 on provider astro 2024-03-08 07:29:50 proc_lsst.multi:146 [INFO] found job 16912296 in provider astro proc_lsst.multi INFO: found job 16912296 in provider astro 2024-03-08 07:29:50 proc_lsst.multi:201 [INFO] cancelling 16912296 on provider astro proc_lsst.multi INFO: cancelling 16912296 on provider astro 2024-03-08 07:29:50 proc_lsst.multi:146 [INFO] found job 16912375 in provider checkpoint proc_lsst.multi INFO: found job 16912375 in provider checkpoint 2024-03-08 07:29:50 proc_lsst.multi:201 [INFO] cancelling 16912375 on provider checkpoint proc_lsst.multi INFO: cancelling 16912375 on provider checkpoint 2024-03-08 07:29:50 proc_lsst.multi:146 [INFO] found job 16912292 in provider astro proc_lsst.multi INFO: found job 16912292 in provider astro 2024-03-08 07:29:50 proc_lsst.multi:201 [INFO] cancelling 16912292 on provider astro proc_lsst.multi INFO: cancelling 16912292 on provider astro 2024-03-08 07:29:50 proc_lsst.multi:146 [INFO] found job 16912298 in provider astro proc_lsst.multi INFO: found job 16912298 in provider astro 2024-03-08 07:29:50 proc_lsst.multi:201 [INFO] cancelling 16912298 on provider astro proc_lsst.multi INFO: cancelling 16912298 on provider astro 2024-03-08 07:29:50 proc_lsst.multi:146 [INFO] found job 16912371 in provider checkpoint proc_lsst.multi INFO: found job 16912371 in provider checkpoint 2024-03-08 07:29:50 proc_lsst.multi:201 [INFO] cancelling 16912371 on provider checkpoint proc_lsst.multi INFO: cancelling 16912371 on provider checkpoint 2024-03-08 07:29:50 proc_lsst.multi:146 [INFO] found job 16912344 in provider checkpoint proc_lsst.multi INFO: found job 16912344 in provider checkpoint 2024-03-08 07:29:50 proc_lsst.multi:201 [INFO] cancelling 16912344 on provider checkpoint proc_lsst.multi INFO: cancelling 16912344 on provider checkpoint 2024-03-08 07:29:51 proc_lsst.multi:146 [INFO] found job 16912322 in provider escience_a40 proc_lsst.multi INFO: found job 16912322 in provider escience_a40 2024-03-08 07:29:51 proc_lsst.multi:201 [INFO] cancelling 16912322 on provider escience_a40 proc_lsst.multi INFO: cancelling 16912322 on provider escience_a40 2024-03-08 07:29:51 proc_lsst.multi:146 [INFO] found job 16912314 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912314 in provider escience_rtx6k 2024-03-08 07:29:51 proc_lsst.multi:201 [INFO] cancelling 16912314 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16912314 on provider escience_rtx6k 2024-03-08 07:29:51 proc_lsst.multi:146 [INFO] found job 16912331 in provider escience_a40 proc_lsst.multi INFO: found job 16912331 in provider escience_a40 2024-03-08 07:29:51 proc_lsst.multi:201 [INFO] cancelling 16912331 on provider escience_a40 proc_lsst.multi INFO: cancelling 16912331 on provider escience_a40 2024-03-08 07:29:51 proc_lsst.multi:146 [INFO] found job 16912312 in provider astro proc_lsst.multi INFO: found job 16912312 in provider astro 2024-03-08 07:29:51 proc_lsst.multi:201 [INFO] cancelling 16912312 on provider astro proc_lsst.multi INFO: cancelling 16912312 on provider astro 2024-03-08 07:29:51 proc_lsst.multi:146 [INFO] found job 16912329 in provider escience_a40 proc_lsst.multi INFO: found job 16912329 in provider escience_a40 2024-03-08 07:29:51 proc_lsst.multi:201 [INFO] cancelling 16912329 on provider escience_a40 proc_lsst.multi INFO: cancelling 16912329 on provider escience_a40 2024-03-08 07:29:51 proc_lsst.multi:146 [INFO] found job 16912333 in provider escience_a40 proc_lsst.multi INFO: found job 16912333 in provider escience_a40 2024-03-08 07:29:51 proc_lsst.multi:201 [INFO] cancelling 16912333 on provider escience_a40 proc_lsst.multi INFO: cancelling 16912333 on provider escience_a40 2024-03-08 07:29:51 proc_lsst.multi:146 [INFO] found job 16912309 in provider astro proc_lsst.multi INFO: found job 16912309 in provider astro 2024-03-08 07:29:51 proc_lsst.multi:201 [INFO] cancelling 16912309 on provider astro proc_lsst.multi INFO: cancelling 16912309 on provider astro 2024-03-08 07:29:51 proc_lsst.multi:146 [INFO] found job 16912325 in provider escience_a40 proc_lsst.multi INFO: found job 16912325 in provider escience_a40 2024-03-08 07:29:51 proc_lsst.multi:201 [INFO] cancelling 16912325 on provider escience_a40 proc_lsst.multi INFO: cancelling 16912325 on provider escience_a40 2024-03-08 07:29:52 proc_lsst.multi:146 [INFO] found job 16912330 in provider escience_a40 proc_lsst.multi INFO: found job 16912330 in provider escience_a40 2024-03-08 07:29:52 proc_lsst.multi:201 [INFO] cancelling 16912330 on provider escience_a40 proc_lsst.multi INFO: cancelling 16912330 on provider escience_a40 2024-03-08 07:29:52 proc_lsst.multi:146 [INFO] found job 16912299 in provider astro proc_lsst.multi INFO: found job 16912299 in provider astro 2024-03-08 07:29:52 proc_lsst.multi:201 [INFO] cancelling 16912299 on provider astro proc_lsst.multi INFO: cancelling 16912299 on provider astro 2024-03-08 07:29:52 proc_lsst.multi:146 [INFO] found job 16912324 in provider escience_a40 proc_lsst.multi INFO: found job 16912324 in provider escience_a40 2024-03-08 07:29:52 proc_lsst.multi:201 [INFO] cancelling 16912324 on provider escience_a40 proc_lsst.multi INFO: cancelling 16912324 on provider escience_a40 2024-03-08 07:29:52 proc_lsst.multi:146 [INFO] found job 16912285 in provider astro proc_lsst.multi INFO: found job 16912285 in provider astro 2024-03-08 07:29:52 proc_lsst.multi:201 [INFO] cancelling 16912285 on provider astro proc_lsst.multi INFO: cancelling 16912285 on provider astro 2024-03-08 07:29:52 proc_lsst.multi:146 [INFO] found job 16912286 in provider astro proc_lsst.multi INFO: found job 16912286 in provider astro 2024-03-08 07:29:52 proc_lsst.multi:201 [INFO] cancelling 16912286 on provider astro proc_lsst.multi INFO: cancelling 16912286 on provider astro 2024-03-08 07:29:52 proc_lsst.multi:146 [INFO] found job 16912290 in provider astro proc_lsst.multi INFO: found job 16912290 in provider astro 2024-03-08 07:29:52 proc_lsst.multi:201 [INFO] cancelling 16912290 on provider astro proc_lsst.multi INFO: cancelling 16912290 on provider astro 2024-03-08 07:29:54 proc_lsst.multi:146 [INFO] found job 16912306 in provider astro proc_lsst.multi INFO: found job 16912306 in provider astro 2024-03-08 07:29:54 proc_lsst.multi:201 [INFO] cancelling 16912306 on provider astro proc_lsst.multi INFO: cancelling 16912306 on provider astro parsl.dataflow.dflow INFO: Shutting down executor multi 2024-03-08 07:29:54 proc_lsst.multi:40 [INFO] Cancelling all provider resources proc_lsst.multi INFO: Cancelling all provider resources 2024-03-08 07:29:54 proc_lsst.multi:47 [INFO] new jobs since last cancel ['16912338', '16912339', '16912340', '16912341', '16912342', '16912343', '16912344', '16912345', '16912346', '16912347', '16912348', '16912349', '16912350', '16912351', '16912352', '16912353', '16912354', '16912355', '16912356', '16912359', '16912360', '16912361', '16912362', '16912363', '16912364', '16912365', '16912366', '16912367', '16912368', '16912369', '16912370', '16912371', '16912372', '16912373', '16912374', '16912375', '16912378', '16912283', '16912284', '16912285', '16912286', '16912287', '16912288', '16912289', '16912290', '16912291', '16912292', '16912293', '16912294', '16912295', '16912296', '16912297', '16912298', '16912299', '16912300', '16912301', '16912302', '16912303', '16912304', '16912305', '16912306', '16912307', '16912308', '16912309', '16912310', '16912311', '16912312', '16912318', '16912319', '16912320', '16912321', '16912322', '16912323', '16912324', '16912325', '16912326', '16912327', '16912328', '16912329', '16912330', '16912331', '16912332', '16912333', '16912334', '16912335', '16912336', '16912337', '16912313', '16912314', '16912315', '16912316', '16912317', '75775'] proc_lsst.multi INFO: new jobs since last cancel ['16912338', '16912339', '16912340', '16912341', '16912342', '16912343', '16912344', '16912345', '16912346', '16912347', '16912348', '16912349', '16912350', '16912351', '16912352', '16912353', '16912354', '16912355', '16912356', '16912359', '16912360', '16912361', '16912362', '16912363', '16912364', '16912365', '16912366', '16912367', '16912368', '16912369', '16912370', '16912371', '16912372', '16912373', '16912374', '16912375', '16912378', '16912283', '16912284', '16912285', '16912286', '16912287', '16912288', '16912289', '16912290', '16912291', '16912292', '16912293', '16912294', '16912295', '16912296', '16912297', '16912298', '16912299', '16912300', '16912301', '16912302', '16912303', '16912304', '16912305', '16912306', '16912307', '16912308', '16912309', '16912310', '16912311', '16912312', '16912318', '16912319', '16912320', '16912321', '16912322', '16912323', '16912324', '16912325', '16912326', '16912327', '16912328', '16912329', '16912330', '16912331', '16912332', '16912333', '16912334', '16912335', '16912336', '16912337', '16912313', '16912314', '16912315', '16912316', '16912317', '75775'] 2024-03-08 07:29:54 proc_lsst.multi:146 [INFO] found job 16912338 in provider checkpoint proc_lsst.multi INFO: found job 16912338 in provider checkpoint 2024-03-08 07:29:54 proc_lsst.multi:201 [INFO] cancelling 16912338 on provider checkpoint proc_lsst.multi INFO: cancelling 16912338 on provider checkpoint 2024-03-08 07:29:55 proc_lsst.multi:146 [INFO] found job 16912339 in provider checkpoint proc_lsst.multi INFO: found job 16912339 in provider checkpoint 2024-03-08 07:29:55 proc_lsst.multi:201 [INFO] cancelling 16912339 on provider checkpoint proc_lsst.multi INFO: cancelling 16912339 on provider checkpoint 2024-03-08 07:29:56 proc_lsst.multi:146 [INFO] found job 16912340 in provider checkpoint proc_lsst.multi INFO: found job 16912340 in provider checkpoint 2024-03-08 07:29:56 proc_lsst.multi:201 [INFO] cancelling 16912340 on provider checkpoint proc_lsst.multi INFO: cancelling 16912340 on provider checkpoint 2024-03-08 07:29:56 proc_lsst.multi:146 [INFO] found job 16912341 in provider checkpoint proc_lsst.multi INFO: found job 16912341 in provider checkpoint 2024-03-08 07:29:56 proc_lsst.multi:201 [INFO] cancelling 16912341 on provider checkpoint proc_lsst.multi INFO: cancelling 16912341 on provider checkpoint 2024-03-08 07:29:56 proc_lsst.multi:146 [INFO] found job 16912342 in provider checkpoint proc_lsst.multi INFO: found job 16912342 in provider checkpoint 2024-03-08 07:29:56 proc_lsst.multi:201 [INFO] cancelling 16912342 on provider checkpoint proc_lsst.multi INFO: cancelling 16912342 on provider checkpoint 2024-03-08 07:29:57 proc_lsst.multi:146 [INFO] found job 16912343 in provider checkpoint proc_lsst.multi INFO: found job 16912343 in provider checkpoint 2024-03-08 07:29:57 proc_lsst.multi:201 [INFO] cancelling 16912343 on provider checkpoint proc_lsst.multi INFO: cancelling 16912343 on provider checkpoint 2024-03-08 07:29:57 proc_lsst.multi:146 [INFO] found job 16912344 in provider checkpoint proc_lsst.multi INFO: found job 16912344 in provider checkpoint 2024-03-08 07:29:57 proc_lsst.multi:201 [INFO] cancelling 16912344 on provider checkpoint proc_lsst.multi INFO: cancelling 16912344 on provider checkpoint 2024-03-08 07:29:57 proc_lsst.multi:146 [INFO] found job 16912345 in provider checkpoint proc_lsst.multi INFO: found job 16912345 in provider checkpoint 2024-03-08 07:29:57 proc_lsst.multi:201 [INFO] cancelling 16912345 on provider checkpoint proc_lsst.multi INFO: cancelling 16912345 on provider checkpoint 2024-03-08 07:29:57 proc_lsst.multi:146 [INFO] found job 16912346 in provider checkpoint proc_lsst.multi INFO: found job 16912346 in provider checkpoint 2024-03-08 07:29:57 proc_lsst.multi:201 [INFO] cancelling 16912346 on provider checkpoint proc_lsst.multi INFO: cancelling 16912346 on provider checkpoint 2024-03-08 07:29:57 proc_lsst.multi:146 [INFO] found job 16912347 in provider checkpoint proc_lsst.multi INFO: found job 16912347 in provider checkpoint 2024-03-08 07:29:57 proc_lsst.multi:201 [INFO] cancelling 16912347 on provider checkpoint proc_lsst.multi INFO: cancelling 16912347 on provider checkpoint 2024-03-08 07:29:57 proc_lsst.multi:146 [INFO] found job 16912348 in provider checkpoint proc_lsst.multi INFO: found job 16912348 in provider checkpoint 2024-03-08 07:29:57 proc_lsst.multi:201 [INFO] cancelling 16912348 on provider checkpoint proc_lsst.multi INFO: cancelling 16912348 on provider checkpoint 2024-03-08 07:29:58 proc_lsst.multi:146 [INFO] found job 16912349 in provider checkpoint proc_lsst.multi INFO: found job 16912349 in provider checkpoint 2024-03-08 07:29:58 proc_lsst.multi:201 [INFO] cancelling 16912349 on provider checkpoint proc_lsst.multi INFO: cancelling 16912349 on provider checkpoint 2024-03-08 07:29:58 proc_lsst.multi:146 [INFO] found job 16912350 in provider checkpoint proc_lsst.multi INFO: found job 16912350 in provider checkpoint 2024-03-08 07:29:58 proc_lsst.multi:201 [INFO] cancelling 16912350 on provider checkpoint proc_lsst.multi INFO: cancelling 16912350 on provider checkpoint 2024-03-08 07:29:58 proc_lsst.multi:146 [INFO] found job 16912351 in provider checkpoint proc_lsst.multi INFO: found job 16912351 in provider checkpoint 2024-03-08 07:29:58 proc_lsst.multi:201 [INFO] cancelling 16912351 on provider checkpoint proc_lsst.multi INFO: cancelling 16912351 on provider checkpoint 2024-03-08 07:29:58 proc_lsst.multi:146 [INFO] found job 16912352 in provider checkpoint proc_lsst.multi INFO: found job 16912352 in provider checkpoint 2024-03-08 07:29:58 proc_lsst.multi:201 [INFO] cancelling 16912352 on provider checkpoint proc_lsst.multi INFO: cancelling 16912352 on provider checkpoint 2024-03-08 07:29:58 proc_lsst.multi:146 [INFO] found job 16912353 in provider checkpoint proc_lsst.multi INFO: found job 16912353 in provider checkpoint 2024-03-08 07:29:58 proc_lsst.multi:201 [INFO] cancelling 16912353 on provider checkpoint proc_lsst.multi INFO: cancelling 16912353 on provider checkpoint 2024-03-08 07:29:59 proc_lsst.multi:146 [INFO] found job 16912354 in provider checkpoint proc_lsst.multi INFO: found job 16912354 in provider checkpoint 2024-03-08 07:29:59 proc_lsst.multi:201 [INFO] cancelling 16912354 on provider checkpoint proc_lsst.multi INFO: cancelling 16912354 on provider checkpoint 2024-03-08 07:29:59 proc_lsst.multi:146 [INFO] found job 16912355 in provider checkpoint proc_lsst.multi INFO: found job 16912355 in provider checkpoint 2024-03-08 07:29:59 proc_lsst.multi:201 [INFO] cancelling 16912355 on provider checkpoint proc_lsst.multi INFO: cancelling 16912355 on provider checkpoint 2024-03-08 07:29:59 proc_lsst.multi:146 [INFO] found job 16912356 in provider checkpoint proc_lsst.multi INFO: found job 16912356 in provider checkpoint 2024-03-08 07:29:59 proc_lsst.multi:201 [INFO] cancelling 16912356 on provider checkpoint proc_lsst.multi INFO: cancelling 16912356 on provider checkpoint 2024-03-08 07:29:59 proc_lsst.multi:146 [INFO] found job 16912359 in provider checkpoint proc_lsst.multi INFO: found job 16912359 in provider checkpoint 2024-03-08 07:29:59 proc_lsst.multi:201 [INFO] cancelling 16912359 on provider checkpoint proc_lsst.multi INFO: cancelling 16912359 on provider checkpoint 2024-03-08 07:29:59 proc_lsst.multi:146 [INFO] found job 16912360 in provider checkpoint proc_lsst.multi INFO: found job 16912360 in provider checkpoint 2024-03-08 07:29:59 proc_lsst.multi:201 [INFO] cancelling 16912360 on provider checkpoint proc_lsst.multi INFO: cancelling 16912360 on provider checkpoint 2024-03-08 07:29:59 proc_lsst.multi:146 [INFO] found job 16912361 in provider checkpoint proc_lsst.multi INFO: found job 16912361 in provider checkpoint 2024-03-08 07:29:59 proc_lsst.multi:201 [INFO] cancelling 16912361 on provider checkpoint proc_lsst.multi INFO: cancelling 16912361 on provider checkpoint 2024-03-08 07:29:59 proc_lsst.multi:146 [INFO] found job 16912362 in provider checkpoint proc_lsst.multi INFO: found job 16912362 in provider checkpoint 2024-03-08 07:29:59 proc_lsst.multi:201 [INFO] cancelling 16912362 on provider checkpoint proc_lsst.multi INFO: cancelling 16912362 on provider checkpoint 2024-03-08 07:29:59 proc_lsst.multi:146 [INFO] found job 16912363 in provider checkpoint proc_lsst.multi INFO: found job 16912363 in provider checkpoint 2024-03-08 07:29:59 proc_lsst.multi:201 [INFO] cancelling 16912363 on provider checkpoint proc_lsst.multi INFO: cancelling 16912363 on provider checkpoint 2024-03-08 07:29:59 proc_lsst.multi:146 [INFO] found job 16912364 in provider checkpoint proc_lsst.multi INFO: found job 16912364 in provider checkpoint 2024-03-08 07:29:59 proc_lsst.multi:201 [INFO] cancelling 16912364 on provider checkpoint proc_lsst.multi INFO: cancelling 16912364 on provider checkpoint 2024-03-08 07:29:59 proc_lsst.multi:146 [INFO] found job 16912365 in provider checkpoint proc_lsst.multi INFO: found job 16912365 in provider checkpoint 2024-03-08 07:29:59 proc_lsst.multi:201 [INFO] cancelling 16912365 on provider checkpoint proc_lsst.multi INFO: cancelling 16912365 on provider checkpoint 2024-03-08 07:29:59 proc_lsst.multi:146 [INFO] found job 16912366 in provider checkpoint proc_lsst.multi INFO: found job 16912366 in provider checkpoint 2024-03-08 07:29:59 proc_lsst.multi:201 [INFO] cancelling 16912366 on provider checkpoint proc_lsst.multi INFO: cancelling 16912366 on provider checkpoint 2024-03-08 07:30:00 proc_lsst.multi:146 [INFO] found job 16912367 in provider checkpoint proc_lsst.multi INFO: found job 16912367 in provider checkpoint 2024-03-08 07:30:00 proc_lsst.multi:201 [INFO] cancelling 16912367 on provider checkpoint proc_lsst.multi INFO: cancelling 16912367 on provider checkpoint 2024-03-08 07:30:00 proc_lsst.multi:146 [INFO] found job 16912368 in provider checkpoint proc_lsst.multi INFO: found job 16912368 in provider checkpoint 2024-03-08 07:30:00 proc_lsst.multi:201 [INFO] cancelling 16912368 on provider checkpoint proc_lsst.multi INFO: cancelling 16912368 on provider checkpoint 2024-03-08 07:30:00 proc_lsst.multi:146 [INFO] found job 16912369 in provider checkpoint proc_lsst.multi INFO: found job 16912369 in provider checkpoint 2024-03-08 07:30:00 proc_lsst.multi:201 [INFO] cancelling 16912369 on provider checkpoint proc_lsst.multi INFO: cancelling 16912369 on provider checkpoint 2024-03-08 07:30:00 proc_lsst.multi:146 [INFO] found job 16912370 in provider checkpoint proc_lsst.multi INFO: found job 16912370 in provider checkpoint 2024-03-08 07:30:00 proc_lsst.multi:201 [INFO] cancelling 16912370 on provider checkpoint proc_lsst.multi INFO: cancelling 16912370 on provider checkpoint 2024-03-08 07:30:00 proc_lsst.multi:146 [INFO] found job 16912371 in provider checkpoint proc_lsst.multi INFO: found job 16912371 in provider checkpoint 2024-03-08 07:30:00 proc_lsst.multi:201 [INFO] cancelling 16912371 on provider checkpoint proc_lsst.multi INFO: cancelling 16912371 on provider checkpoint 2024-03-08 07:30:00 proc_lsst.multi:146 [INFO] found job 16912372 in provider checkpoint proc_lsst.multi INFO: found job 16912372 in provider checkpoint 2024-03-08 07:30:00 proc_lsst.multi:201 [INFO] cancelling 16912372 on provider checkpoint proc_lsst.multi INFO: cancelling 16912372 on provider checkpoint 2024-03-08 07:30:00 proc_lsst.multi:146 [INFO] found job 16912373 in provider checkpoint proc_lsst.multi INFO: found job 16912373 in provider checkpoint 2024-03-08 07:30:00 proc_lsst.multi:201 [INFO] cancelling 16912373 on provider checkpoint proc_lsst.multi INFO: cancelling 16912373 on provider checkpoint 2024-03-08 07:30:00 proc_lsst.multi:146 [INFO] found job 16912374 in provider checkpoint proc_lsst.multi INFO: found job 16912374 in provider checkpoint 2024-03-08 07:30:00 proc_lsst.multi:201 [INFO] cancelling 16912374 on provider checkpoint proc_lsst.multi INFO: cancelling 16912374 on provider checkpoint 2024-03-08 07:30:00 proc_lsst.multi:146 [INFO] found job 16912375 in provider checkpoint proc_lsst.multi INFO: found job 16912375 in provider checkpoint 2024-03-08 07:30:00 proc_lsst.multi:201 [INFO] cancelling 16912375 on provider checkpoint proc_lsst.multi INFO: cancelling 16912375 on provider checkpoint 2024-03-08 07:30:00 proc_lsst.multi:146 [INFO] found job 16912378 in provider checkpoint proc_lsst.multi INFO: found job 16912378 in provider checkpoint 2024-03-08 07:30:00 proc_lsst.multi:201 [INFO] cancelling 16912378 on provider checkpoint proc_lsst.multi INFO: cancelling 16912378 on provider checkpoint 2024-03-08 07:30:00 proc_lsst.multi:146 [INFO] found job 16912283 in provider astro proc_lsst.multi INFO: found job 16912283 in provider astro 2024-03-08 07:30:00 proc_lsst.multi:201 [INFO] cancelling 16912283 on provider astro proc_lsst.multi INFO: cancelling 16912283 on provider astro 2024-03-08 07:30:01 proc_lsst.multi:146 [INFO] found job 16912284 in provider astro proc_lsst.multi INFO: found job 16912284 in provider astro 2024-03-08 07:30:01 proc_lsst.multi:201 [INFO] cancelling 16912284 on provider astro proc_lsst.multi INFO: cancelling 16912284 on provider astro 2024-03-08 07:30:01 proc_lsst.multi:146 [INFO] found job 16912285 in provider astro proc_lsst.multi INFO: found job 16912285 in provider astro 2024-03-08 07:30:01 proc_lsst.multi:201 [INFO] cancelling 16912285 on provider astro proc_lsst.multi INFO: cancelling 16912285 on provider astro 2024-03-08 07:30:01 proc_lsst.multi:146 [INFO] found job 16912286 in provider astro proc_lsst.multi INFO: found job 16912286 in provider astro 2024-03-08 07:30:01 proc_lsst.multi:201 [INFO] cancelling 16912286 on provider astro proc_lsst.multi INFO: cancelling 16912286 on provider astro 2024-03-08 07:30:01 proc_lsst.multi:146 [INFO] found job 16912287 in provider astro proc_lsst.multi INFO: found job 16912287 in provider astro 2024-03-08 07:30:01 proc_lsst.multi:201 [INFO] cancelling 16912287 on provider astro proc_lsst.multi INFO: cancelling 16912287 on provider astro 2024-03-08 07:30:01 proc_lsst.multi:146 [INFO] found job 16912288 in provider astro proc_lsst.multi INFO: found job 16912288 in provider astro 2024-03-08 07:30:01 proc_lsst.multi:201 [INFO] cancelling 16912288 on provider astro proc_lsst.multi INFO: cancelling 16912288 on provider astro 2024-03-08 07:30:01 proc_lsst.multi:146 [INFO] found job 16912289 in provider astro proc_lsst.multi INFO: found job 16912289 in provider astro 2024-03-08 07:30:01 proc_lsst.multi:201 [INFO] cancelling 16912289 on provider astro proc_lsst.multi INFO: cancelling 16912289 on provider astro 2024-03-08 07:30:01 proc_lsst.multi:146 [INFO] found job 16912290 in provider astro proc_lsst.multi INFO: found job 16912290 in provider astro 2024-03-08 07:30:01 proc_lsst.multi:201 [INFO] cancelling 16912290 on provider astro proc_lsst.multi INFO: cancelling 16912290 on provider astro 2024-03-08 07:30:01 proc_lsst.multi:146 [INFO] found job 16912291 in provider astro proc_lsst.multi INFO: found job 16912291 in provider astro 2024-03-08 07:30:01 proc_lsst.multi:201 [INFO] cancelling 16912291 on provider astro proc_lsst.multi INFO: cancelling 16912291 on provider astro 2024-03-08 07:30:01 proc_lsst.multi:146 [INFO] found job 16912292 in provider astro proc_lsst.multi INFO: found job 16912292 in provider astro 2024-03-08 07:30:01 proc_lsst.multi:201 [INFO] cancelling 16912292 on provider astro proc_lsst.multi INFO: cancelling 16912292 on provider astro 2024-03-08 07:30:02 proc_lsst.multi:146 [INFO] found job 16912293 in provider astro proc_lsst.multi INFO: found job 16912293 in provider astro 2024-03-08 07:30:02 proc_lsst.multi:201 [INFO] cancelling 16912293 on provider astro proc_lsst.multi INFO: cancelling 16912293 on provider astro 2024-03-08 07:30:02 proc_lsst.multi:146 [INFO] found job 16912294 in provider astro proc_lsst.multi INFO: found job 16912294 in provider astro 2024-03-08 07:30:02 proc_lsst.multi:201 [INFO] cancelling 16912294 on provider astro proc_lsst.multi INFO: cancelling 16912294 on provider astro 2024-03-08 07:30:02 proc_lsst.multi:146 [INFO] found job 16912295 in provider astro proc_lsst.multi INFO: found job 16912295 in provider astro 2024-03-08 07:30:02 proc_lsst.multi:201 [INFO] cancelling 16912295 on provider astro proc_lsst.multi INFO: cancelling 16912295 on provider astro 2024-03-08 07:30:02 proc_lsst.multi:146 [INFO] found job 16912296 in provider astro proc_lsst.multi INFO: found job 16912296 in provider astro 2024-03-08 07:30:02 proc_lsst.multi:201 [INFO] cancelling 16912296 on provider astro proc_lsst.multi INFO: cancelling 16912296 on provider astro 2024-03-08 07:30:02 proc_lsst.multi:146 [INFO] found job 16912297 in provider astro proc_lsst.multi INFO: found job 16912297 in provider astro 2024-03-08 07:30:02 proc_lsst.multi:201 [INFO] cancelling 16912297 on provider astro proc_lsst.multi INFO: cancelling 16912297 on provider astro 2024-03-08 07:30:02 proc_lsst.multi:146 [INFO] found job 16912298 in provider astro proc_lsst.multi INFO: found job 16912298 in provider astro 2024-03-08 07:30:02 proc_lsst.multi:201 [INFO] cancelling 16912298 on provider astro proc_lsst.multi INFO: cancelling 16912298 on provider astro 2024-03-08 07:30:02 proc_lsst.multi:146 [INFO] found job 16912299 in provider astro proc_lsst.multi INFO: found job 16912299 in provider astro 2024-03-08 07:30:02 proc_lsst.multi:201 [INFO] cancelling 16912299 on provider astro proc_lsst.multi INFO: cancelling 16912299 on provider astro 2024-03-08 07:30:02 proc_lsst.multi:146 [INFO] found job 16912300 in provider astro proc_lsst.multi INFO: found job 16912300 in provider astro 2024-03-08 07:30:02 proc_lsst.multi:201 [INFO] cancelling 16912300 on provider astro proc_lsst.multi INFO: cancelling 16912300 on provider astro 2024-03-08 07:30:02 proc_lsst.multi:146 [INFO] found job 16912301 in provider astro proc_lsst.multi INFO: found job 16912301 in provider astro 2024-03-08 07:30:02 proc_lsst.multi:201 [INFO] cancelling 16912301 on provider astro proc_lsst.multi INFO: cancelling 16912301 on provider astro 2024-03-08 07:30:02 proc_lsst.multi:146 [INFO] found job 16912302 in provider astro proc_lsst.multi INFO: found job 16912302 in provider astro 2024-03-08 07:30:02 proc_lsst.multi:201 [INFO] cancelling 16912302 on provider astro proc_lsst.multi INFO: cancelling 16912302 on provider astro 2024-03-08 07:30:02 proc_lsst.multi:146 [INFO] found job 16912303 in provider astro proc_lsst.multi INFO: found job 16912303 in provider astro 2024-03-08 07:30:02 proc_lsst.multi:201 [INFO] cancelling 16912303 on provider astro proc_lsst.multi INFO: cancelling 16912303 on provider astro 2024-03-08 07:30:03 proc_lsst.multi:146 [INFO] found job 16912304 in provider astro proc_lsst.multi INFO: found job 16912304 in provider astro 2024-03-08 07:30:03 proc_lsst.multi:201 [INFO] cancelling 16912304 on provider astro proc_lsst.multi INFO: cancelling 16912304 on provider astro 2024-03-08 07:30:03 proc_lsst.multi:146 [INFO] found job 16912305 in provider astro proc_lsst.multi INFO: found job 16912305 in provider astro 2024-03-08 07:30:03 proc_lsst.multi:201 [INFO] cancelling 16912305 on provider astro proc_lsst.multi INFO: cancelling 16912305 on provider astro 2024-03-08 07:30:03 proc_lsst.multi:146 [INFO] found job 16912306 in provider astro proc_lsst.multi INFO: found job 16912306 in provider astro 2024-03-08 07:30:03 proc_lsst.multi:201 [INFO] cancelling 16912306 on provider astro proc_lsst.multi INFO: cancelling 16912306 on provider astro 2024-03-08 07:30:03 proc_lsst.multi:146 [INFO] found job 16912307 in provider astro proc_lsst.multi INFO: found job 16912307 in provider astro 2024-03-08 07:30:03 proc_lsst.multi:201 [INFO] cancelling 16912307 on provider astro proc_lsst.multi INFO: cancelling 16912307 on provider astro 2024-03-08 07:30:03 proc_lsst.multi:146 [INFO] found job 16912308 in provider astro proc_lsst.multi INFO: found job 16912308 in provider astro 2024-03-08 07:30:03 proc_lsst.multi:201 [INFO] cancelling 16912308 on provider astro proc_lsst.multi INFO: cancelling 16912308 on provider astro 2024-03-08 07:30:03 proc_lsst.multi:146 [INFO] found job 16912309 in provider astro proc_lsst.multi INFO: found job 16912309 in provider astro 2024-03-08 07:30:03 proc_lsst.multi:201 [INFO] cancelling 16912309 on provider astro proc_lsst.multi INFO: cancelling 16912309 on provider astro 2024-03-08 07:30:03 proc_lsst.multi:146 [INFO] found job 16912310 in provider astro proc_lsst.multi INFO: found job 16912310 in provider astro 2024-03-08 07:30:03 proc_lsst.multi:201 [INFO] cancelling 16912310 on provider astro proc_lsst.multi INFO: cancelling 16912310 on provider astro 2024-03-08 07:30:03 proc_lsst.multi:146 [INFO] found job 16912311 in provider astro proc_lsst.multi INFO: found job 16912311 in provider astro 2024-03-08 07:30:03 proc_lsst.multi:201 [INFO] cancelling 16912311 on provider astro proc_lsst.multi INFO: cancelling 16912311 on provider astro 2024-03-08 07:30:03 proc_lsst.multi:146 [INFO] found job 16912312 in provider astro proc_lsst.multi INFO: found job 16912312 in provider astro 2024-03-08 07:30:03 proc_lsst.multi:201 [INFO] cancelling 16912312 on provider astro proc_lsst.multi INFO: cancelling 16912312 on provider astro 2024-03-08 07:30:03 proc_lsst.multi:146 [INFO] found job 16912318 in provider escience_a40 proc_lsst.multi INFO: found job 16912318 in provider escience_a40 2024-03-08 07:30:03 proc_lsst.multi:201 [INFO] cancelling 16912318 on provider escience_a40 proc_lsst.multi INFO: cancelling 16912318 on provider escience_a40 2024-03-08 07:30:03 proc_lsst.multi:146 [INFO] found job 16912319 in provider escience_a40 proc_lsst.multi INFO: found job 16912319 in provider escience_a40 2024-03-08 07:30:03 proc_lsst.multi:201 [INFO] cancelling 16912319 on provider escience_a40 proc_lsst.multi INFO: cancelling 16912319 on provider escience_a40 2024-03-08 07:30:04 proc_lsst.multi:146 [INFO] found job 16912320 in provider escience_a40 proc_lsst.multi INFO: found job 16912320 in provider escience_a40 2024-03-08 07:30:04 proc_lsst.multi:201 [INFO] cancelling 16912320 on provider escience_a40 proc_lsst.multi INFO: cancelling 16912320 on provider escience_a40 2024-03-08 07:30:04 proc_lsst.multi:146 [INFO] found job 16912321 in provider escience_a40 proc_lsst.multi INFO: found job 16912321 in provider escience_a40 2024-03-08 07:30:04 proc_lsst.multi:201 [INFO] cancelling 16912321 on provider escience_a40 proc_lsst.multi INFO: cancelling 16912321 on provider escience_a40 2024-03-08 07:30:04 proc_lsst.multi:146 [INFO] found job 16912322 in provider escience_a40 proc_lsst.multi INFO: found job 16912322 in provider escience_a40 2024-03-08 07:30:04 proc_lsst.multi:201 [INFO] cancelling 16912322 on provider escience_a40 proc_lsst.multi INFO: cancelling 16912322 on provider escience_a40 2024-03-08 07:30:04 proc_lsst.multi:146 [INFO] found job 16912323 in provider escience_a40 proc_lsst.multi INFO: found job 16912323 in provider escience_a40 2024-03-08 07:30:04 proc_lsst.multi:201 [INFO] cancelling 16912323 on provider escience_a40 proc_lsst.multi INFO: cancelling 16912323 on provider escience_a40 2024-03-08 07:30:04 proc_lsst.multi:146 [INFO] found job 16912324 in provider escience_a40 proc_lsst.multi INFO: found job 16912324 in provider escience_a40 2024-03-08 07:30:04 proc_lsst.multi:201 [INFO] cancelling 16912324 on provider escience_a40 proc_lsst.multi INFO: cancelling 16912324 on provider escience_a40 2024-03-08 07:30:04 proc_lsst.multi:146 [INFO] found job 16912325 in provider escience_a40 proc_lsst.multi INFO: found job 16912325 in provider escience_a40 2024-03-08 07:30:04 proc_lsst.multi:201 [INFO] cancelling 16912325 on provider escience_a40 proc_lsst.multi INFO: cancelling 16912325 on provider escience_a40 2024-03-08 07:30:04 proc_lsst.multi:146 [INFO] found job 16912326 in provider escience_a40 proc_lsst.multi INFO: found job 16912326 in provider escience_a40 2024-03-08 07:30:04 proc_lsst.multi:201 [INFO] cancelling 16912326 on provider escience_a40 proc_lsst.multi INFO: cancelling 16912326 on provider escience_a40 2024-03-08 07:30:04 proc_lsst.multi:146 [INFO] found job 16912327 in provider escience_a40 proc_lsst.multi INFO: found job 16912327 in provider escience_a40 2024-03-08 07:30:04 proc_lsst.multi:201 [INFO] cancelling 16912327 on provider escience_a40 proc_lsst.multi INFO: cancelling 16912327 on provider escience_a40 2024-03-08 07:30:04 proc_lsst.multi:146 [INFO] found job 16912328 in provider escience_a40 proc_lsst.multi INFO: found job 16912328 in provider escience_a40 2024-03-08 07:30:04 proc_lsst.multi:201 [INFO] cancelling 16912328 on provider escience_a40 proc_lsst.multi INFO: cancelling 16912328 on provider escience_a40 2024-03-08 07:30:04 proc_lsst.multi:146 [INFO] found job 16912329 in provider escience_a40 proc_lsst.multi INFO: found job 16912329 in provider escience_a40 2024-03-08 07:30:04 proc_lsst.multi:201 [INFO] cancelling 16912329 on provider escience_a40 proc_lsst.multi INFO: cancelling 16912329 on provider escience_a40 2024-03-08 07:30:04 proc_lsst.multi:146 [INFO] found job 16912330 in provider escience_a40 proc_lsst.multi INFO: found job 16912330 in provider escience_a40 2024-03-08 07:30:04 proc_lsst.multi:201 [INFO] cancelling 16912330 on provider escience_a40 proc_lsst.multi INFO: cancelling 16912330 on provider escience_a40 2024-03-08 07:30:05 proc_lsst.multi:146 [INFO] found job 16912331 in provider escience_a40 proc_lsst.multi INFO: found job 16912331 in provider escience_a40 2024-03-08 07:30:05 proc_lsst.multi:201 [INFO] cancelling 16912331 on provider escience_a40 proc_lsst.multi INFO: cancelling 16912331 on provider escience_a40 2024-03-08 07:30:05 proc_lsst.multi:146 [INFO] found job 16912332 in provider escience_a40 proc_lsst.multi INFO: found job 16912332 in provider escience_a40 2024-03-08 07:30:05 proc_lsst.multi:201 [INFO] cancelling 16912332 on provider escience_a40 proc_lsst.multi INFO: cancelling 16912332 on provider escience_a40 2024-03-08 07:30:05 proc_lsst.multi:146 [INFO] found job 16912333 in provider escience_a40 proc_lsst.multi INFO: found job 16912333 in provider escience_a40 2024-03-08 07:30:05 proc_lsst.multi:201 [INFO] cancelling 16912333 on provider escience_a40 proc_lsst.multi INFO: cancelling 16912333 on provider escience_a40 2024-03-08 07:30:05 proc_lsst.multi:146 [INFO] found job 16912334 in provider escience_a40 proc_lsst.multi INFO: found job 16912334 in provider escience_a40 2024-03-08 07:30:05 proc_lsst.multi:201 [INFO] cancelling 16912334 on provider escience_a40 proc_lsst.multi INFO: cancelling 16912334 on provider escience_a40 2024-03-08 07:30:05 proc_lsst.multi:146 [INFO] found job 16912335 in provider escience_a40 proc_lsst.multi INFO: found job 16912335 in provider escience_a40 2024-03-08 07:30:05 proc_lsst.multi:201 [INFO] cancelling 16912335 on provider escience_a40 proc_lsst.multi INFO: cancelling 16912335 on provider escience_a40 2024-03-08 07:30:05 proc_lsst.multi:146 [INFO] found job 16912336 in provider escience_a40 proc_lsst.multi INFO: found job 16912336 in provider escience_a40 2024-03-08 07:30:05 proc_lsst.multi:201 [INFO] cancelling 16912336 on provider escience_a40 proc_lsst.multi INFO: cancelling 16912336 on provider escience_a40 2024-03-08 07:30:05 proc_lsst.multi:146 [INFO] found job 16912337 in provider escience_a40 proc_lsst.multi INFO: found job 16912337 in provider escience_a40 2024-03-08 07:30:05 proc_lsst.multi:201 [INFO] cancelling 16912337 on provider escience_a40 proc_lsst.multi INFO: cancelling 16912337 on provider escience_a40 2024-03-08 07:30:05 proc_lsst.multi:146 [INFO] found job 16912313 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912313 in provider escience_rtx6k 2024-03-08 07:30:05 proc_lsst.multi:201 [INFO] cancelling 16912313 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16912313 on provider escience_rtx6k 2024-03-08 07:30:05 proc_lsst.multi:146 [INFO] found job 16912314 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912314 in provider escience_rtx6k 2024-03-08 07:30:05 proc_lsst.multi:201 [INFO] cancelling 16912314 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16912314 on provider escience_rtx6k 2024-03-08 07:30:05 proc_lsst.multi:146 [INFO] found job 16912315 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912315 in provider escience_rtx6k 2024-03-08 07:30:05 proc_lsst.multi:201 [INFO] cancelling 16912315 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16912315 on provider escience_rtx6k 2024-03-08 07:30:05 proc_lsst.multi:146 [INFO] found job 16912316 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912316 in provider escience_rtx6k 2024-03-08 07:30:05 proc_lsst.multi:201 [INFO] cancelling 16912316 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16912316 on provider escience_rtx6k 2024-03-08 07:30:06 proc_lsst.multi:146 [INFO] found job 16912317 in provider escience_rtx6k proc_lsst.multi INFO: found job 16912317 in provider escience_rtx6k 2024-03-08 07:30:06 proc_lsst.multi:201 [INFO] cancelling 16912317 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16912317 on provider escience_rtx6k 2024-03-08 07:30:06 proc_lsst.multi:146 [INFO] found job 75775 in provider local proc_lsst.multi INFO: found job 75775 in provider local 2024-03-08 07:30:06 proc_lsst.multi:201 [INFO] cancelling 75775 on provider local proc_lsst.multi INFO: cancelling 75775 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 75775 parsl.providers.local.local WARNING: Failed to kill PID: 75775 and child processes on local 2024-03-08 07:30:07 proc_lsst.multi:50 [INFO] no new jobs since last cancel, resuming executor shutdown proc_lsst.multi INFO: no new jobs since last cancel, resuming executor shutdown parsl.executors.high_throughput.executor INFO: Attempting HighThroughputExecutor shutdown parsl.executors.high_throughput.executor INFO: Finished HighThroughputExecutor shutdown attempt parsl.dataflow.dflow INFO: Shut down executor multi parsl.dataflow.dflow INFO: Shutting down executor _parsl_internal parsl.executors.threads DEBUG: Shutting down executor, which involves waiting for running tasks to complete parsl.executors.threads DEBUG: Done with executor shutdown parsl.dataflow.dflow INFO: Shut down executor _parsl_internal parsl.dataflow.dflow INFO: Terminated executors parsl.dataflow.dflow INFO: DFK cleanup complete parsl.process_loggers DEBUG: Normal ending for cleanup on thread MainThread lsst.ctrl.bps.submit INFO: Completed submitting to a workflow management system: Took 923.1110 seconds lsst.ctrl.bps.drivers INFO: Run 'DEEP_20190504_A0b_science#step1_20240308T071422Z' submitted for execution with id 'None' lsst.ctrl.bps.drivers INFO: Completed submit stage: Took 923.1180 seconds; current memory usage: 0.391 Gibyte, delta: 0.009 Gibyte, peak delta: 0.009 Gibyte lsst.ctrl.bps.drivers INFO: Completed entire submission process: Took 942.6896 seconds; current memory usage: 0.391 Gibyte, delta: 0.255 Gibyte, peak delta: 0.255 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.391 Gibyte (main), 0.391 Gibyte (largest child process) Run Id: None Run Name: DEEP_20190504_A0b_science#step1_20240308T071422Z parsl.dataflow.dflow INFO: python process is exiting, but DFK has already been cleaned up real 15m45.094s user 3m23.281s sys 2m7.540s